Troubleshooting Common Issues

Cannot Reach Server

  1. Ping the server by Hostname and IP Address:

    • Hostname/IP Address is pingable:

      • The issue might be on the client side since the server is reachable.

    • Hostname is not pingable but IP Address is pingable:

      • Likely a DNS issue. Check:

        • /etc/hosts

        • /etc/resolv.conf

        • /etc/nsswitch.conf

        • Test DNS Resolution:

          • Using nslookup, dig or host

    • Neither Hostname nor IP Address is pingable:

      • Check another server on the same network:

        • False: Issue is with this specific host/server.

        • True: Likely a broader network issue.

      • Log in via Virtual Console (if the server is powered on):

        • Check uptime using command uptime.

        • Verify if the server has an IP and if the network interface is UP.

          • Run the command ip addr

          • Ensure the network interface (e.g., eth0, ens33) is listed and in the "UP" state.

        • Ping the gateway and check routes.

        • Check SELinux and firewall rules.

        • Inspect physical cable connections.

Cannot Reach Website or Application

  1. Ping the server by Hostname and IP Address:

    • False: Follow troubleshooting steps from “Server is not reachable or unable to connect.”

    • True: Check service availability using the telnet command with the appropriate port:

      • True: The service is running.

      • False: The service is not reachable or running. Check:

        • Service status (using systemctl or equivalent commands).

        • Firewall/SELinux settings.

        • Service logs.

        • Service configuration.

Unable to SSH as Root or User

  1. Ping the server by Hostname and IP Address:

    • False: Follow troubleshooting steps from “Cannot Reach Server”

    • True: Check service availability using the telnet command with the SSH port:

      • True: The service is running:

        • Check if the issue is on the client side.

        • Verify:

          • User account is not disabled.

          • User has a valid shell (not nologin).

          • Root login is not disabled in the SSH configuration.

      • False: The service is not reachable or running. Check:

        • Service status (using systemctl or equivalent commands).

        • Firewall/SELinux settings.

        • Service logs.

        • Service configuration.

Disk Space is Full or Adding/Extending Disk Space

  1. Detect Performance Degradation:

    • Applications are slow or unresponsive.

    • Commands fail to execute (e.g., / disk space is full).

    • Logging and other system operations fail.

  2. Analyze the Issue:

    • Use the df command to identify the problematic filesystem.

  3. Take Action:

    • Use du to find large files/directories in the affected filesystem.

    • Compress or remove large files.

    • Move files to another partition or server.

    • Check disk health with badblocks (e.g., badblocks -v /dev/sda).

    • Identify I/O-bound processes using iostat.

    • Create a link to move large files/directories.

  4. Add a New Disk:

    • Simple Partition:

      • Add the disk to the VM.

      • Verify the new disk using df or lsblk.

      • Use fdisk to create a partition (preferably LVM).

      • Create a filesystem, mount it, and add it to fstab for persistence.

    • LVM Partition:

      • Add the disk to the VM.

      • Verify with df or lsblk.

      • Use fdisk to create an LVM partition.

      • Set up PV, VG, and LV.

      • Create a filesystem, mount it, and add it to fstab.

    • Extend LVM Partition:

      • Add and create an LVM partition.

      • Add the new LVM partition (PV) to the existing VG.

      • Extend the LV and resize the filesystem.

Filesystem Corruption

  1. Symptoms:

    • The system fails to boot.

  2. Check Logs:

    • Investigate /var/log/messages, dmesg, and other log files.

    • Look for bad sector logs.

  3. Run fsck if Bad Sectors are Found:

    • Reboot the system into rescue mode (e.g., boot from CD-ROM or ISO).

    • Select Option 1 to mount the original root filesystem under /mnt/sysimage.

    • Edit fstab entries or recreate the file using blkid.

    • Reboot the system.

Missing or Incorrect fstab File

  1. Symptoms:

    • The system fails to boot.

  2. Check Logs:

    • Investigate /var/log/messages, dmesg, and other log files.

    • Look for bad sector logs.

  3. Run fsck if Bad Sectors are Found:

    • Reboot the system into rescue mode (e.g., boot from CD-ROM or ISO).

    • Select Option 1 to mount the original root filesystem under /mnt/sysimage.

    • Edit fstab entries or recreate the file using blkid.

    • Reboot the system.

Cannot cd to Directory (Even with Sudo Privileges)

  1. Reasons and Resolutions:

    • Directory does not exist.

    • Pathname conflict (relative vs absolute path).

    • Parent directory permission or ownership issues.

    • Missing executable permissions on the target directory.

    • Hidden directory not visible.

Cannot Create Links

  1. Reasons and Resolutions:

    • Target directory or file does not exist.

    • Pathname conflict (relative vs absolute path) — ensure the path is complete.

    • Parent directory permission or ownership issues.

    • Target file permission or ownership issues — must have read permissions.

    • Hidden directory or file not visible.

Running Out of Memory

  1. Types of Memory:

    • Cache: L1, L2, L3.

    • RAM:

      • Usage details from free -h:

        • Total: Total assigned memory.

        • Used: Total memory actually in use.

        • Free: Memory available for immediate use.

        • Shared: Shared memory.

        • Buff/Cache: Pages cached in memory.

        • Available: Memory that can be freed.

      • Check /proc/meminfo for detailed metrics:

        • File active/inactive, Anon active/inactive.

    • Swap (Virtual Memory): Monitor and manage for system stability.

  2. Resolutions:

    • Identify high-memory processes using top, htop, or ps.

    • Check logs for OOM events and review memory overcommit settings in sysctl.conf.

    • Kill or restart memory-hogging processes/services.

    • Use nice to prioritize critical processes.

    • Add or extend swap space.

    • Install more physical RAM.

Add or Extend Swap Space

  1. Steps to Add Swap Space:

    • Create a file using dd to reserve disk blocks for swap.

    • Set file permissions to 600 and assign root ownership.

    • Format the file for swap with mkswap.

    • Enable swap using swapon.

    • Add the swap file to fstab for persistence.

Unable to Run Certain Commands

  1. Troubleshooting and Resolutions:

    • Command issues:

      • System-related commands may require root access.

      • User-defined scripts/commands might have restrictions.

    • Steps to troubleshoot:

      • Check permission or ownership of the command/script.

      • Ensure sudo privileges are configured.

      • Verify the absolute or relative path to the command/script.

      • Ensure the command is in the user's $PATH variable.

      • Confirm that the command is installed.

      • Check for missing or deleted command libraries.

System Unexpectedly Rebooting and Processes Restarting

  1. Troubleshooting and Resolution:

    • System Reboot/Crash Reasons:

      • CPU stress.

      • RAM stress.

      • Kernel fault.

      • Hardware fault.

    • Process Restart Causes:

      • System reboot triggers process restarts.

      • Processes might restart themselves.

      • Watchdog applications:

        • Prevent high stress on system resources.

        • Restart or terminate processes causing excessive stress.

    • Troubleshooting Steps:

      • After logging in, check system status using commands like:

        • uptime, top, dmesg, journalctl, iostat -xz 1.

      • Examine log files: syslog.log, boot.log, dmesg, messages.log.

      • Check custom application log paths.

      • If inaccessible, use virtual consoles (e.g., ILO, IDRAC).

      • Open a support case with the vendor if needed.

Unable to Get an IP Address

  1. IP Assignment Methods:

    • DHCP:

      • Fixed Allocation.

      • Dynamic Allocation.

    • Static IP.

  2. Troubleshooting Steps:

    • Check network settings in the virtualization environment (e.g., VMware, VirtualBox).

    • Verify whether an IP address has been assigned.

    • Check the NIC status on the host using tools like lspci, nmcli.

    • Restart the network service.

Backup and Restore File Permissions in Linux

  1. Backup and Restore Steps:

    • The best option is to create an ACL file for directories/files before making bulk permission changes:

      • Backup file permissions: getfacl -R <dir> > permissions.acl.

      • Restore file permissions: setfacl --restore=permissions.acl.

    • Restore using a VM snapshot (not ideal for production environments).

    • Rebuild the VM (a safer option for long-term stability).

Useful Tips Related to Disk Partitioning

  1. Tips for Managing Disk Partitions:

    • After attaching a new disk to a VM, use lsblk to check its status, then rescan using:

      • echo 1 > /sys/block/sda/device/rescan.

    • Increasing the size of an existing disk appends additional space to the disk without affecting the existing file system or partition.

    • Recreating the filesystem on a block device automatically formats the old one.

    • For a disk with an existing partition/filesystem, share the .vmdk file to another VM. After mounting, the data will remain identical.

PreviousSplitting a Monolith into MicroservicesNextService Level Terminology

Last updated

Was this helpful?