Search this Blog

Thursday, January 30, 2014

How can we retrieve core files from Cisco Nexus switching platforms ?

What are the  common steps and commands used to retrieve core files in Nexus switching platforms -  Nexus7000, Nexus5000, Nexus 4000, Nexus 3000 and Nexus2000.

When a specific process (called as Service) crahes, the device should report a log message, as follows:

Scenario 1:

%SYSMGR-2-SERVICE_CRASHED: Service "vpc" (PID 5883) hasn't caught signal 11 (core will be saved)

Here, service "vpc" has crashed and a core file will be saved.

Scenario 2:

The device may report message, with no core file created.

%SYSMGR-2-SERVICE_CRASHED: Service "stp" (PID 4668) hasn't caught signal 9 (no core).

Here, sevice "stp" crashed but has not generated any core file.

Retreiving Core files:

For Scenario 1 (as mentioned above):

If there is a process crash/exception reported and the switch has NOT reloaded (since the exception/crash), then do "show cores" to get list of cores.

N7K# show cores
VDC Module Instance Process-name     PID       Date(Year-Month-Day Time)
--- ------ -------- --------------- -------- -------------------------
1   6       1         vpc             4763     2011-01-10 11:33:01
1   6       1         vpc             5883     2011-01-10 11:33:05

Please do "show cores vdc-all" to see core files in all VDCs.

The above results indicate that the exception was reported for "vpc" service in VDC #1, Module #6.
The results provide different Process ID (PID) - 4763 and 5883 - the specific process had at exception, with timestamps.
Instance number will be useful to identify the core files when a specific process with same PID (for the same VDC) experience multiple exceptions.

Please be aware that "show cores" command do NOT provide any information,  if the switch has rebooted since the exception.

To copy the core files to FTP or TFTP server, follow the steps:

N7K# copy core:?
   core: Enter URL "core:///[/instance-num]“

N7K# copy core://6/4763/1 ?
   bootflash: Select destination filesystem
   ftp:       Select destination filesystem
   scp:       Select destination filesystem
   sftp:      Select destination filesystem
   slot0:     Select destination filesystem
   tftp:      Select destination filesystem
   usb1:      Select destination filesystem
   usb2:      Select destination filesystem

The above command collects all relevant info (system info, log files etc.) from the switch and bundles them into .tar file.
It is NOT recommended to copy files directly from different filesystems manually.

If the switch has rebooted, do following command to see if there are core files generated earlier:

N7K# dir logflash://sup-1/core 
100499456   Aug 29 22:36:54 2011 0x501_ethpm_core.16574
   8638991   Aug 29 22:45:14 2011 0x501_ethpm_core.4165.gz
     37139   Aug 29 22:36:54 2011 0x501_ethpm_log.16574
   7699061   Aug 29 22:36:32 2011 0x501_ethpm_log.16576.tar.gz
   8208542   Aug 29 22:36:32 2011 0x501_ethpm_log.4165.tar.gz
   7698622   Aug 29 22:45:30 2011 1314657930_0x501_ethpm_log.16576.tar.gz
   8208230   Aug 29 22:45:30 2011 1314657930_0x501_ethpm_log.4165.tar.gz

If there is Supervisor failover occurred, please check the other/standby sup for core files.

N7K# dir logflash://sup-2/core

In Nexus5000, Nexus4000, Nexus3000 and Nexus2000 platforms, as there is no supervisor engine redundancy, there will not be any failover.


In Nexus5000, Nexus4000, Nexus3000 and Nexus2000 platforms the core files are stored in the "volatile:" and not in the "logflash:" file system.

N3k-3# dir volatile:?

Please be aware that contents of "volatile:" file system are flushed on reload.

For Scenario 2 (as mentioned above):

N7K# show process log vdc-all
VDC Process         PID     Normal-exit Stack Core   Log-create-time
--- --------------- ------ ----------- ----- ----- ---------------
1 installer       10544             N     N     N Thu Jun 10 17:49:21 2010
1 ethpm           16574             N     Y     N Mon Aug 29 22:36:15 2011

Here, the "ethpm" sevice crashed and generated "Stack" (flagged with Y) but no "Core" file (flagged with N).
At the same, for the "installer" process, neither "Stack" nor "Core" file is generated.

For the "installer" process, furher information can be obtained by:

N7K# show process log pid 10544
Service: installer
Description: Installer
Started at Thu Jun 10 17:45:42 2010 (483528 us)
Stopped at Thu Jun 10 17:49:21 2010 (719259 us)
Uptime: 3 minutes 39 seconds
Last heartbeat 0.00 secs ago
RLIMIT_AS: 69909875
System image name: n7000-s1-dk9.4.2.4.bin
System image version: 4.2(4) S32
PID: 10544
SAP: 0

For the "ethpm" process, the stack trace can be obtained by:

N7K# show process log pid 16574
Service: ethpm
Description: Test Ethernet Port Manager
Executable: /isan/bin/ethpm
Started at Mon Aug 29 22:36:15 2011 (188136 us)
Stopped at Mon Aug 29 22:36:15 2011 (746741 us)
Uptime: 0 seconds
Virtual Memory:
   CODE     08048000 - 08356C90
   DATA     08357000 - 08369BA8
   BRK       083F0000 - 086F9000
   STACK     BFBB25C0
   TOTAL     98996 KB
Memory Map: 08048000 ethp 08357000 ethp 4143F000 ld-2.8.s 41459000 ld-2.8.s 4145
A000 ld-2.8.s 4145D000 libc-2.8.s 41596000 libc-2.8.s 41598000 libc-2.8.s 4159E0
Register Set:
   EBX BFBB0ADC         ECX 00000000         EDX 00000002
   ESI BFBB15B0         EDI 00000009         EBP BFBB1148
Stack: 6976 bytes. ESP BFBB0A80, TOP BFBB25C0
0xBFBB0A80: 0000001F 00000000 00000000 00000001 ................

 Why the core file is missing ? :

If the switch does not have enough space in the specific filesystem (logflash: or volatile: depending on the platform), then the core file may not be successfully generated/stored.

N7K# dir logflash://sup-1/
Usage for logflash://sup-1
  498237440 bytes used
7394926592 bytes free
7893164032 bytes total

To check the free space available in different file systems, you can also do:

N7K# show system internal flash
Mount-on                  1K-blocks      Used   Available   Use%  Filesystem
/                            409600     61372      348228     15   /dev/root
/proc                             0         0           0      0   proc
/sys                              0         0           0      0   none
/isan                       1048576    339184      709392     33   none
/bootflash                  1809684    673252     1044504     40   /dev/hda3
/logflash                   7708168     95004     7221608      2   /dev/hde1
/bootflash_sup-remote       1809688    672952     1044808     40
/logflash_sup-remote        7708168     34976     7281640      1

Same set of commands, from a Nexus3000 switch:

N3K# dir volatile://sup-1/
Usage for volatile://sup-1
          0 bytes used
  104857600 bytes free
  104857600 bytes total

N3K# sh system internal flash
Mount-on                  1K-blocks      Used   Available   Use%  Filesystem
/                            204800    112436       92364     55   /dev/root
/proc                             0         0           0      0   proc
/post                          2048         4        2044      1   none
/sys                              0         0           0      0   none
/volatile                    102400         0      102400      0   none
/debug                        20480         8       20472      1   none
/bootflash                  1609984    582492      945708     39   /dev/sda3

 Logs/Files to Capture:

If further analysis required on process exception / core files, please open a Service Request and send following logs:

- show cores vdc-all
- Core files saved using "copy core:///[/instance-num]..." command
- show process log vdc-all
- show process log details
- show logging onboard internal reset-reason
- show logging onboard stack-trace
- show logging onboard kernel-trace
- show module internal exceptionlog module

Please make sure all these logs are captured to a file(s), as the logs may go several pages.

Citation - This blog post does not reflect original content from the author. Rather it summarizes content that are relevant to the topic from different sources in the web. The sources might include any online discussion boards, forums, websites and others.

1 comment :

  1. I found your this post while searching for some related information on blog search...Its a good post..keep posting and update the information recover deleted files from USB


/* Google Analytics begin ----------------------------------------------- */ /* Google Analytics end ----------------------------------------------- */