AX 2012 - Errors in workflow - Due to issues in CIL (ContractFilter mismatch at the EndpointDispatcher, or SysWorkflowConfigurationService issue)

In AX 2012, the workflow can stop working if there is a mess up in XPPIL (path: C:\Program Files\Microsoft Dynamics AX\60\Server\[AOS-Instance-Name]\bin\XppIL)  folder and SYSXPPASSEMBLY table (where actual IL assemblies are stored) in Model DB, which might not be matching with each other. This can result in unexpected behavior not only in Workflows but other AOT frameworks dependent on the SysOperation framework which depends upon perfectly running IL. It is very important that there should not be any native error in the whole AOT, then only CIL could compile successfully. The SysOperation services will run successfully and the endpoint will be listening after successful CIL compilation.

I have faced one scenario when MS Partner ends up in failure without resolving such an issue after 2 weeks of struggle, the handed over to MS support which never started. However, on the investigation, it appeared that IL assembly in DB was not matching with physical files in XPPIL due to the copy/pasting approach followed by MS partner, where the physical files were at least 2 months before, that too may not be belonging to the same server. They might have resolved the BIService to run SSRS, but the workflow framework was messed up.

The issue was reported in workflow history as Stopped without any description. On checking the Event Log of Windows, it showed as below:

Workflow Instance ID: 065634 mscorlib The message with Action 'http://tempuri.org/SysWorkflowConfigurationService/getConfiguration' cannot be processed at the receiver, due to a ContractFilter mismatch at the EndpointDispatcher. This may be because of either a contract mismatch (mismatched Actions between sender and receiver) or a binding/security mismatch between the sender and the receiver.  Check that sender and receiver have the same contract and the same binding (including security requirements, e.g. Message, Transport, None). 

Server stack trace: 

   at System.ServiceModel.Channels.ServiceChannel.ThrowIfFaultUnderstood(Message reply, MessageFault fault, String action, MessageVersion version, FaultConverter faultConverter)

   at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc)

   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)

   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)

   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)



Exception rethrown at [0]: 

   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)

   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)

   at Microsoft.Dynamics.AX.Framework.Workflow.Model.AxWorkflowServiceReference.SysWorkflowConfigurationService.getConfiguration(SysWorkflowConfigurationServiceGetConfigurationRequest request)

   at Microsoft.Dynamics.AX.Framework.Workflow.Model.AxServiceClient.getConfiguration(Guid _configurationId, String _companyId)

   at Microsoft.Dynamics.AX.Framework.Workflow.Model.WorkflowModel.LoadModel(Int64 workflowRecId, Guid workflowId, String companyId, Boolean runtime, String partitionKey)

   at Microsoft.Dynamics.AX.Framework.Workflow.Model.WorkflowModel.Create(Guid workflowId, Boolean runtime)

   at Microsoft.Dynamics.AX.Framework.Workflow.Runtime.AxWorkflowService.GetConfiguration(Guid configurationId, String companyId, String domainUser)

   at Microsoft.Dynamics.AX.Framework.Workflow.Runtime.WorkflowInstanceManager.Execute(WorkflowMessage message, Guid instanceId, String instanceNumber, String originator, String submitter, Guid affinity)

On further investigation, the workflow configuration form was also not opening and throwing some error without details. Then to catch inner exception following code was amended in WorkflowEditorHost form, in build method (as highlighted) :

\Forms\WorkflowEditorHost\Methods\build

private void build()

....

    System.Exception            interopException;

 ............

    else

    {

        try

        {

            workflowConfiguration = Microsoft.Dynamics.AX.Framework.Workflow.Model.WorkflowModel::Create(versionTable.ConfigurationId, curext(), domainUser);

        }

        catch (Exception::CLRError)

        {

            interopException = CLRInterop::getLastException();

            while (!CLRInterop::isNull(interopException.get_InnerException()))

            {

                interopException = interopException.get_InnerException();

            }

            error(strFmt("%1", CLRInterop::getAnyTypeForObject(interopException.get_Message())));

            throw error("@SYS327400");

        }

    }

}

}

After editing workflow configuration, now the inner exception was somewhat:

Could not find endpoint element with name 'NetTcpBinding_SysWorkflowConfigurationService' and contract 'Microsoft.Dynamics.AX.Framework.Services.Client.AxClient.SysWorkflowConfigurationService' in the ServiceModel client configuration section. This might be because no configuration file was found for your application, or because no endpoint element matching this name could be found in the client element.

This was obviously related with SysWorkflowConfigurationService which is linked with AxClient Service Group, and manual deployment was not successful due to underlying issue of CIL mess up or mismatching IL assemblies in DB vs files. This was also true on checking AX Client configuration *.axc file which should have an entry as below. Contrary to that it was not found even after freshly generated WCF without such entry as AXClient SysOperation service group was not successfully deployed:

<endpoint address="net.tcp://[aoserver]:8201/DynamicsAx/Services/AxClient" binding="netTcpBinding" bindingConfiguration="NetTcpBinding_SysWorkflowConfigurationService" contract="Microsoft.Dynamics.AX.Framework.Services.Client.AxClient.SysWorkflowConfigurationService" name="NetTcpBinding_SysWorkflowConfigurationService">

To resolve the above issues, one should follow the below sequence:

1. Compile Natively without any errors.

2. Truncate the SYSXPPASSEMBLY table in model DB and remove everything in XPPIL (path: C:\Program Files\Microsoft Dynamics AX\60\Server\[AOS-Instance-Name]\bin\XppIL).

3. Compile CIL fully to generate new assemblies.

4. This would automatically deploy AXClient service group. If not, you should try manually deploying the same from AOT.

5. Finally, generate fresh WCF and generate a fresh client configuration file, to open AX Client. Also, check the above mentioned end point entry if it was generated which also means AXClient Service Group successfully deployed.

Finally the above mentioned issue in workflow should be resolved, if the batch jobs for workflow are running without issues. Also, workflow configuration form should be opened without any issues if the issue was resolved.