Oracle 12c Agent Blocked due to Plugin Mismatch [NOT_FOUND_IN_INPUT]

I recently ran into this problem when Oracle Management Agent 12c was in blocked state and was showing below errors:

Run below query as SYSMAN on repository database. Provide hostname of blocked agent when prompted:

SQL> select * from mgmt_blocked_agents where TARGET_GUID in (select target_guid from mgmt_targets where lower(target_name) like '%&hostname%');

TARGET_GUID                      BLOCKED_T

-------------------------------- ---------

BLOCKED_BY

------------------------------------------------------------------------------------------------------------------------------------------------------

BLOCKED_REASON_MSG

------------------------------------------------------------------------------------------------------------------------------------------------------

BLOCKED_REASON_NLS_ID

----------------------------------------------------------------

BLOCKED_REASON_NLS_PARAMS

------------------------------------------------------------------------------------------------------------------------------------------------------

016FFE88BD675EE4DBD5E5ACD779314B 08-MAR-17

SYSMAN

Plugin mismatches between agent and repository. Check plugin:[ The plug-in is oracle.sysman.db

Version in pla inventory : 12.1.0.2.0

Version on agent : null

Content type : AGENT

Error code : NOT_FOUND_IN_INPUT

Error Msg : This plug-in is present in the inventory for the given agent but has not been provided in the input,  The plug-in is oracle.sysman.oh

Version in pla inventory : 12.1.0.2.0

Version on agent : null

Content type : AGENT

Error code : NOT_FOUND_IN_INPUT

Error Msg : This plug-in is present in the inventory for the given agent but has not been provided in the input,  The plug-in is oracle.sysman.db

Version in pla inventory : 12.1.0.2.0

Version on agent : null

Content type : DISCOVERY

Error code : NOT_FOUND_IN_INPUT

Error Msg : This plug-in is present in the inventory for the given agent but has not been provided in the input,  The plug-in is oracle.sysman.oh

Version in pla inventory : 12.1.0.2.0

Version on agent : null

Content type : DISCOVERY

Error code : NOT_FOUND_IN_INPUT

Error Msg : This plug-in is present in the inventory for the given agent but has not been provided in the input]

BLK_AGENT_FOR_MISMATCH

The problem actually started when agent had gone out-of-sync from repository:

INFO - Reason the OMS blocked the agent: Agent is out-of-sync with repository. This most likely means that the agent was reinstalled or recovered. Please contact an EM administrator to unblock the agent by performing an agent resync from the console.

This should be resolved by re-synchronizing the agent via OEM or command-line:

1. Using OEM Console
   Navigate to All Targets > Agent Running on Host >Agent Menu > Resynchronization
   This would submit the job for resync

OR

2. Using command-line
   Set environment for OMS on host where OMS is running
   $ emcli login -username=SYSMAN
   $ emcli sync
   $ emcli resyncAgent -Agent="Agent_Hostname:Port"

However, in my case the resync resulted in success but still the agent was not unblocked and errors were still appearing in EM.

I went ahead and investigated resync logfiles on agent host under "/<Agent_Dir>/agent_inst/install/logs" and found

below errors in: agentplugindeploy_29.log
--------------------------------------------------------
The attachhome /usr/oracle/agent12c/core/12.1.0.2.0/oui/bin/runInstaller  -attachHome -silent -waitforcompletion
-invPtrLoc /usr/oracle/agent12c/core/12.1.0.2.0/oraInst.loc ORACLE_HOME=/usr/oracle/agent12c/core/12.1.0.2.0/../../plugins/oracle.sysman.oh.discovery.plugin_12.1.0.2.0
ORACLE_HOME_NAME=oracle_sysman_oh_12_1_0_2_0_discovery_Home4 "HOME_DEPENDENCY_LIST={/usr/oracle/agent12c/core/12.1.0.2.0/../../plugins/oracle.sysman.oh.discovery.plugin_12.1.0.2.0:/usr/oracle/agent12c/core/12.1.0.2.0}" 
-force -ignoreSysPrereqs   failed : trying for 1 time  return value is : 65280
--------------------------------------------------------
 
and more errors in: agentplugindeploy_29.log.err
--------------------------------------------------------
You do not have sufficient permissions to access the inventory '/usr/oracle/11.2.0.1'.
Installation cannot continue. Make sure that you have read/write permissions to the inventory directory and restart the installer.:
The file access permissions do not allow the specified action.
--------------------------------------------------------

This error indicates that resync operation is not able to access local inventory of an Oracle Home (/usr/oracle/11.2.0.1). The home was actually empty and owned by root:system. This must have been left out due to improper deinstallation of old home.

$ ls -a /usr/oracle/11.2.0.1

.   ..

$ ls -ld /usr/oracle/11.2.0.1

drwxr-xr-x    2 root     system          256 29 Nov 2012  /usr/oracle/11.2.0.1

Since the ownership was not correct and the directory wasn't accessible, this needed to be fixed.

$ chown oracle:dba /usr/oracle/11.2.0.1

$ ls -ld /usr/oracle/11.2.0.1

drwxr-xr-x    2 oracle     dba          256 29 Nov 2012  /usr/oracle/11.2.0.1

I re-submitted resync job and the problem was resolved.

Cheers!
Anurag