Hi,
I am having this issue with my local cluster. I checked the partition is not an issue I have more than 10GB available on all nodes in cluster. I am trying to deploy MS Dynamics 365FO on the local cluster according to this doc: https://docs.microsoft.com/en-us/dynamics365/unified-operations/dev-itpro/deployment/setup-deploy-on-premises-pu12#plansvcacct and i am stuck on the last step where i have deploy to the local server.
Error in the SF:
_______________________________________
Error event: SourceId='System.FM', Property='State'. Partition is below target replica or instance count. fabric:/AXSF/AXService -1 1 76897cf8-fba9-4968-9517-ab9d80816b5e IB AOS_83 Up 131703551926567717 IB AOS_91 Up 131703551796481193 IB AOS_80 Up 131703551939536215 (Showing 3 out of 3 replicas. Total available replicas: 0) For more information see: http://aka.ms/sfhealth
___________________________________
Error on each node:
Unhealthy event: SourceId='System.RA', Property='ReplicaOpenStatus', HealthState='Warning', ConsiderWarningAsError=false. Replica had multiple failures during open on AOS_80. API call: IStatelessServiceInstance.Open(); Error = System.Exception (-2146233088) DB sync failed. at Microsoft.Dynamics.AXService.Database.Synchronizer.SyncDB() at Microsoft.Dynamics.AXService.Database.Synchronizer.Synchronize() at Microsoft.Dynamics.AXService.AXService.<>c__DisplayClass9_1.<.ctor>b__0() at Microsoft.PowerApps.Runtime.Common.LatencyRecorder.RecordLatencyEvent(ILogger logger, String eventName, Action action, IDictionary2 additionalProperties, IDictionary2 additionalMetrics) at Microsoft.Dynamics.AXService.AXService..ctor(StatelessServiceContext context, ILogger logger) at Microsoft.Dynamics.AXService.Program.<>c.b__0_0(StatelessServiceContext context) at Microsoft.ServiceFabric.Services.Runtime.StatelessServiceInstanceFactory.System.Fabric.IStatelessServiceFactory.CreateInstance(String serviceTypeName, Uri serviceName, Byte[] initializationData, Guid partitionId, Int64 instanceId) at System.Fabric.ServiceFactoryBroker.<>c__DisplayClass9_0.b__0(IStatelessServiceFactory factory, ServiceInitializationParameters initParams) at System.Fabric.ServiceFactoryBroker.CreateHelper[TFactory,TReturnValue](IntPtr nativeServiceType, IntPtr nativeServiceName, UInt32 initializationDataLength, IntPtr nativeInitializationData, Guid partitionId, Func3 creationFunc, Action2 initializationFunc, ServiceInitializationParameters initializationParameters) For more information see: http://aka.ms/sfhealth
_______________________________________________________
Errors in the diagnostic store for the node where its failing:
2018/05/07-15:39:21.914,Info,4524,FabricDeployer.FabricDeployer,Running deployer with UpdateInstanceId /fabricBinRoot:C:\Program Files\Microsoft Service Fabric\bin /fabricDataRoot:C:\ProgramData\SF /fabricLogRoot:C:\ProgramData\SF\Log /cm:C:\ProgramData\SF\AOS_83\Fabric\ClusterManifest.current.xml /oldClusterManifestString: /im:C:\ProgramData\SF\AOS_83\Fabric\Fabric.Data\InfrastructureManifest.xml /instanceId:131701810926192303 /targetVersion:6.1.480.9494 /nodeName:AOS_83 /nodeTypeName: /runAsType: /runAsAccountName: /runAsPassword: /serviceStartupType: /output: /currentVersion: /error:C:\ProgramData\SF\AOS_83\Fabric\work\41-3164 /bootstrapMSIPath: /machineName: /fabricPackageRoot: /jsonClusterConfigLocation: /enableCircularTraceSession: /continueIfContainersFeatureNotInstalled: /skipDeleteData:
2018/05/07-15:39:21.945,Info,4524,ImageStoreClient.ManagedFileLock,Obtained writer lock for C:\ProgramData\SF\lock
2018/05/07-15:39:21.945,Info,4524,FabricDeployer.FabricDeployer,Executing UpdateInstanceId /fabricBinRoot:C:\Program Files\Microsoft Service Fabric\bin /fabricDataRoot:C:\ProgramData\SF /fabricLogRoot:C:\ProgramData\SF\Log /cm:C:\ProgramData\SF\AOS_83\Fabric\ClusterManifest.current.xml /oldClusterManifestString: /im:C:\ProgramData\SF\AOS_83\Fabric\Fabric.Data\InfrastructureManifest.xml /instanceId:131701810926192303 /targetVersion:6.1.480.9494 /nodeName:AOS_83 /nodeTypeName: /runAsType: /runAsAccountName: /runAsPassword: /serviceStartupType: /output: /currentVersion: /error:C:\ProgramData\SF\AOS_83\Fabric\work\41-3164 /bootstrapMSIPath: /machineName: /fabricPackageRoot: /jsonClusterConfigLocation: /enableCircularTraceSession: /continueIfContainersFeatureNotInstalled: /skipDeleteData:
2018/05/07-15:39:22.117,Info,4524,FabricDeployer.FabricDeployer,Running operation System.Fabric.FabricDeployer.UpdateInstanceIdOperation
2018/05/07-15:39:22.148,Info,4524,FabricDeployer.FabricDeployer,AclClusterLevelCerts: Enter
2018/05/07-15:39:22.148,Info,4524,FabricDeployer.FabricDeployer,AclClusterLevelCerts: processing FindByThumbprint,43E5DDB7BE39DDCD37B3EF508FD597AAD709F475,My
2018/05/07-15:39:22.164,Info,4524,FabricDeployer.FabricDeployer,AclCert success: FindByThumbprint,43E5DDB7BE39DDCD37B3EF508FD597AAD709F475,My
2018/05/07-15:39:22.164,Info,4524,FabricDeployer.FabricDeployer,AclClusterLevelCerts: FindByThumbprint,43E5DDB7BE39DDCD37B3EF508FD597AAD709F475,My processed
2018/05/07-15:39:22.179,Info,4524,FabricDeployer.FabricDeployer,Skipping container network set up since we are not on Azure.
2018/05/07-15:39:22.601,Info,4524,FabricDeployer.FabricDeployer,The current machine IP addresses are: fe80::ec51:5049:948e:e189%5, 192.168.145.83, 127.0.0.1, ::1
2018/05/07-15:39:22.695,Info,4524,FabricDeployer.FabricDeployer,ServiceFabricEtlFileQueryable section was not found and is being added.
2018/05/07-15:39:22.711,Info,4524,FabricDeployer.FabricDeployer,ServiceFabricEtlFileQueryable section was not found and is being added.
2018/05/07-15:39:23.148,Info,4524,ImageStoreClient.ManagedFileLock,Obtained writer lock for C:\ProgramData\SF\AOS_83\Fabric\work\UnreliableTransportSettings.ini
2018/05/07-15:39:23.164,Info,4524,ImageStoreClient.ManagedFileLock,Released writer lock on C:\ProgramData\SF\AOS_83\Fabric\work\UnreliableTransportSettings.ini
2018/05/07-15:39:23.226,Info,4524,FabricDeployer.FabricDeployer,Generating the fabric package file at C:\ProgramData\SF\AOS_83\Fabric\Fabric.Package.current.xml
2018/05/07-15:39:23.257,Info,4524,FabricDeployer.FabricDeployer,Host Settings file generated at C:\ProgramData\SF\FabricHostSettings.xml
2018/05/07-15:39:26.711,Info,4524,FabricDeployer.FabricDeployer,Dynamic port range not specified in cluster manifest. Start port : 0. Using default dynamic port range for server.
2018/05/07-15:39:26.726,Info,4524,FabricDeployer.FabricDeployer,Executing command: logman stop FabricCounters
2018/05/07-15:39:29.143,Info,4524,FabricDeployer.FabricDeployer,Executing command: logman delete FabricCounters
2018/05/07-15:39:29.670,Info,4524,FabricDeployer.FabricDeployer,Performance counter sampling interval: 00:01:00 seconds.
2018/05/07-15:39:29.686,Info,4524,FabricDeployer.FabricDeployer,Executing command: logman create counter FabricCounters -cf C:\Windows\TEMP\tmp2BDC.tmp -f bin -si 60 -o "C:\ProgramData\SF\Log\PerformanceCountersBinary\fabric_counters_636612899696865084" -v nnnnnn -max 50 -cnf 2700
2018/05/07-15:39:30.249,Info,4524,FabricDeployer.FabricDeployer,Executing command: logman start FabricCounters
2018/05/07-15:39:31.853,Info,4524,FabricDeployer.FabricDeployer,Infra node <AOS_80> params.NodeName <AOS_83>
2018/05/07-15:39:31.853,Info,4524,FabricDeployer.FabricDeployer,Infra node <AOS_83> params.NodeName <AOS_83>
2018/05/07-15:39:31.853,Info,4524,FabricDeployer.FabricDeployer,Found node IPAddressOrFQDN <192.168.145.83>
2018/05/07-15:39:31.853,Info,4524,FabricDeployer.FabricDeployer,DockerDnsHelper current node IPAddressOrFQDN <192.168.145.83>
2018/05/07-15:39:31.869,Info,4524,FabricDeployer.FabricDeployer,Starting DockerDnsHelper.SetupAsync
2018/05/07-15:39:31.869,Info,4524,FabricDeployer.FabricDeployer,Executing DockerDnsHelper.SetupAsync, attempt 1 of 3
2018/05/07-15:39:31.884,Info,4524,FabricDeployer.FabricDeployer,Dism initializing...
2018/05/07-15:39:31.884,Info,4524,FabricDeployer.FabricDeployer,Dism initializing completed, opening online session...
2018/05/07-15:39:32.279,Info,4524,FabricDeployer.FabricDeployer,Dism opening online session completed, getting feature info...
2018/05/07-15:39:33.666,Info,4524,FabricDeployer.FabricDeployer,Dism feature Containers is in state: DismStateStaged
2018/05/07-15:39:33.666,Info,4524,FabricDeployer.FabricDeployer,Dism closing session...
2018/05/07-15:39:33.791,Info,4524,FabricDeployer.FabricDeployer,Dism closing session completed
2018/05/07-15:39:33.791,Info,4524,FabricDeployer.FabricDeployer,Dism shutting down...
2018/05/07-15:39:33.885,Info,4524,FabricDeployer.FabricDeployer,Dism shutting down completed
2018/05/07-15:39:33.902,Info,4524,FabricDeployer.FabricDeployer,Dism log file content:
2018-05-07 11:39:31, Warning DISM DISM Provider Store: PID=4108 TID=4048 Failed to Load the provider: C:\Windows\SYSTEM32\Dism\SiloedPackageProvider.dll. - CDISMProviderStore::Internal_GetProvider(hr:0x8007007e)
[4108] [0x8007007b] FIOReadFileIntoBuffer:(1250): The filename, directory name, or volume label syntax is incorrect.
[4108] [0xc142011c] UnmarshallImageHandleFromDirectory:(793)
[4108] [0xc142011c] WIMGetMountedImageHandle:(3047)
[4108] [0x8007007b] FIOReadFileIntoBuffer:(1250): The filename, directory name, or volume label syntax is incorrect.
[4108] [0xc142011c] UnmarshallImageHandleFromDirectory:(793)
[4108] [0xc142011c] WIMGetMountedImageHandle:(3047)
2018-05-07 11:39:32, Warning DISM DISM OS Provider: PID=4364 TID=2704 Unable to set the DLL search path to the servicing stack folder. C:\Windows may not point to a valid Windows folder. - CDISMOSServiceManager::Final_OnConnect
2018-05-07 11:39:32, Warning DISM DISM Provider Store: PID=4364 TID=2704 Failed to Load the provider: C:\Windows\TEMP\F578182C-D732-4F74-A6D8-1EAAEF177735\PEProvider.dll. - CDISMProviderStore::Internal_GetProvider(hr:0x8007007e)
2018/05/07-15:39:33.947,Info,4524,FabricDeployer.FabricDeployer,Containers feature not present. Not continuing further
2018/05/07-15:39:33.947,Info,4524,FabricDeployer.FabricDeployer,Successfully completed DockerDnsHelper.SetupAsync
2018/05/07-15:39:33.947,Info,4524,FabricDeployer.FabricDeployer,Completed DockerDnsHelper.SetupAsync
2018/05/07-15:39:33.947,Info,4524,ImageStoreClient.ManagedFileLock,Released writer lock on C:\ProgramData\SF\lock
___________________________________________________
Please let me know if you need more information. I tried to deploy 3-4 times from LCS but it always fail on this step when it deploy AXSF service.