"Offline" agent

A computer status of "Offline" or "Managed (Offline)" means that the Deep Security Manager hasn't communicated with the Deep Security Agent's instance for some time and has exceeded the missed heartbeat threshold. (See Configure the heartbeat.) The status change can also appear in alerts and events.

Causes

Heartbeat connections can fail because:

  • The agent is installed on a workstation or other computer that has been shut down. If you are using Deep Security to protect computers that sometimes get shut down, make sure the policy assigned to those computers does not raise an alert when there is a missed heartbeat. In the policy editor, go to Settings > General > Number of Heartbeats that can be missed before an alert is raised and change the setting to "Unlimited".
  • Firewall, IPS rules, or security groups block the heartbeat port number.
  • Outbound (ephemeral) ports were blocked accidentally. See Blocked port for troubleshooting tips.
  • Bi-directional communication is enabled, but only one direction is allowed or reliable (see Configure communication directionality).
  • Computer is powered off.
  • Computer has left the context of the private network
    This can occur if roaming endpoints (such as a laptop) cannot connect to the manager at their current location. Guest Wi-Fi, for example, often restricts open ports, and has NAT when traffic goes across the Internet.
  • Amazon WorkSpace computer is being powered off, and the heartbeat interval is fast, for example, one minute; in this case, wait until the WorkSpace is fully powered off, and at that point, the status should change from 'Offline' to 'VM Stopped'.
  • DNS was down, or could not resolve the manager's hostname.
  • The manager, the agent, or both are under very high system resource load.
  • The agent process might not be running.
  • Certificates for mutual authentication in the SSL or TLS connection have become invalid or revoked (see Replace the Deep Security Manager TLS certificate).
  • The agent's or manager's system time is incorrect (required by SSL/TLS connections).
  • Deep Security rule update is not yet complete, temporarily interrupting connectivity.
  • On AWS EC2, ICMP traffic is required, but is blocked.
  • After upgrading to agent version 20.0.0.6313 or later, if the agent is still using SHA-1 algorithm. Now the agent only allows newer, more secure cryptographic algorithms for communication to the manager.
If you are using manager-initiated or bi-directional communication, and are having communication issues, we strongly recommend that you change to agent-initiated activation (see Activate and protect agents using agent-initiated activation and communication).

To troubleshoot the error, verify that the agent is running, and then that it can communicate with the manager.

Verify that the agent is running

On the computer with the agent, verify that the Trend Micro Deep Security Agent service is running. Method varies by operating system.

  • On Windows, open the Microsoft Windows Services Console (services.msc) or Task Manager. Look for the service named ds_agent.

  • On Linux, open a terminal and enter the command for a process listing. Look for the service named ds_agent or ds-agent, such as:

    sudo ps -aux | grep ds_agent

    sudo service ds_agent status

  • On Solaris, open a terminal and enter the command for a process listing. Look for the service named ds_agent, such as:

    sudo ps -ef | grep ds_agent

    sudo svcs -l svc:/application/ds_agent:default

Verify DNS

If agents connect to the manager via its domain name or hostname, not its IP address, test the DNS resolution:

nslookup [manager domain name]

DNS service must be reliable.

If the test fails, verify that the agent is using the correct DNS proxy or server (internal domain names can't be resolved by a public DNS server such as Google or your ISP). If a name such as dsm.example.com cannot be resolved into its IP address, communication will fail, even though correct routes and firewall policies exist for the IP address.

If the computer uses DHCP, in the computer or policy settings, in the Advanced Network Engine area, you might need to enable Force Allow DHCP DNS(see Network engine settings).

Allow outbound ports (agent-initiated heartbeat)

Telnet to required port numbers on the manager to verify that a route exists, and the port is open:

telnet [manager IP]:4120

Telnet success proves most of the same things as a ping: that a route and correct firewall policy exist, and that Ethernet frame sizes are correct. (Ping is disabled on computers that use the default security policy for the manager. Networks sometimes block ICMP ping and traceroute to block attackers' reconnaissance scans. So usually, you can't ping the manager to test.)

If telnet fails, trace the route to discover which point on the network is interrupting connectivity.

  • On Linux, enter the command:

    traceroute [agent IP]

  • On Windows, enter the command:

    tracert [agent IP]

Adjust firewall policies, routes, NAT port forwarding, or all three to correct the problem. Verify both network and host-based firewalls, such as Windows Firewall and Linux iptables. For an AWS EC2 instance, see Amazon's documentation on Amazon EC2 Security Groups for Linux Instances or Amazon EC2 Security Groups for Windows Instances. For an Azure VM instance, see Microsoft's Azure documentation on modifying a Network Security Group.

If connectivity tests from the agent to the manager succeed, then next you must test connectivity in the other direction. (Firewalls and routers often require policy-route pairs to allow connectivity. If only 1 of the 2 required policies or routes exist, then packets will be allowed in one direction but not the other.)

Allow inbound ports (manager-initiated heartbeat)

On the manager, ping the agent and telnet to the heartbeat port number to verify that heartbeat and configuration traffic can reach the agent:

ping [agent IP]

telnet [agent IP]:4118

If the ping and telnet fail, use:

traceroute [agent IP]

to discover which point on the network is interrupting connectivity. Adjust firewall policies, routes, NAT port forwarding, or all three to correct the problem.

If IPS or firewall rules are blocking the connection between the agent and the manager, then the manager cannot connect in order to unassign the policy that is causing the problem. To solve this, enter the command on the computer to reset policies on the agent:

dsa_control -r

You must re-activate the agent after running this command.

Allow ICMP on Amazon AWS EC2 instances

In the AWS cloud, routers require ICMP type 3 code 4. If this traffic is blocked, connectivity between agents and the manager may be interrupted.

You can force allow this traffic in Deep Security. Either create a firewall policy with a force allow, or in the computer or policy settings, in the Advanced Network Engine area, enable Force Allow ICMP type3 code4 (see Network engine settings).

Fix the upgrade issue on Solaris 11

A problem may occur if you previously installed Deep Security Agent 9.0 on Solaris 11, and then upgraded the agent software to 11.0 directly without first installing 9.0.0-5616 or a later 9.0 agent. In this scenario, the agent may fail to start up after the upgrade and may appear as offline in Deep Security Manager. To fix this issue:

  1. Uninstall the agent from the server. See Uninstall Deep Security Agent.
  2. Install the Deep Security Agent 11.0. See Install the agent manually.
  3. Re-activate the agent on the manager. See Activate the agent.