Dynamics GP will not remove hold or why timestamp format in GP matters

If Dynamics GP will not let a SOP module, transaction process hold be removed in the user interface, it may be due to formatting of time stamps. Check no integrating applications have created that hold incorrectly. 

The issue is that GP is real fussy about formats of timestamps, one place time stamps can cause problems is in use with the SOP Transaction Hold table, the following query illustrates what you need to know about this. 

SELECT [PRCHLDID], [HOLDDATE], [TIME1]
FROM SOP10104
WHERE SOPNUMBE='YourDocumentNumber' AND SOPTYPE=2

This query gives us some data to look at, that is if some holds exist for that document supplied to the SQL query...

The first thing to notice is that GP often splits/spreads the Time and the Date components of the point in time over two fields, in this case the point in time is represented by HOLDDATE and TIME1. These two fields are of the sql data type "datetime", something that can catch out people when new to GP databases as it may not be totally obvious what is going on. 

Lets look at the first row, it is for the hold type "CREDIT" and represents the point in time, 3rd of May 2020 at eleven minutes past three and 31 seconds

  • It is interesting and important to note that for the HOLDDATE field, it has a date but see how the date has zeros for all the time elements of that HOLDDATE datetime value. To fill up the unused time component, the time has been defaulted to midnight. A default time of zero (midnight) is used, as the unused half of the datetime sql data type may not be set to null (we need all those bytes). 
  • Now also note how the TIME1 field value has a time component, but the date has been defaulted to a "default" of the 1st of January 1900. This date by convention is used to fill the date component of the time, as the unused half of a datetime data type may not null. 
  • Further, it is vital to note that the milliseconds of the time has been zeroed too

Now if you disobey these format rules, the GP user interface will do weird things. More often than not, it just doesn't do anything, that can be really perplexing as a problem solve because the records to the untrained eye look fine and similar to other records right next to them. It may be as simple as including milliseconds (not zeroing them) that can prevent a user from removing a hold in the GP user interface!

These time stamps are used in this way for many tables, where the same rules apply. It is very important to provide a correctly formatted time and date field if integrating or manually scripting against the GP database. This is a really common rookie mistake for developers working with GP, its easy to get lazy or not notice the formatting to be wrong. It can also change behaviour and totals, as time span calculations may not result in the same outcome if time components are left in date fields etc. Imagine the impact when totalling values over time spans, urgh.  

So to save some head scratching, and to make a GP compatible timestamp for the current time field in .NET use the following code snippet:

DateTime now = new DateTime(1900, 1, 1).Add(DateTime.Now.TimeOfDay);
DateTime ForFieldUse= now - new TimeSpan(0, 0, 0, 0, now.TimeOfDay.Milliseconds);

Don't be tempted to simplify this and try to access the .now in two places on the same statement as you may find the times have shifted enough between calls to cause the milliseconds not to zero correctly as two different values would be returned and be subtracted, not resulting in zero! Date times are immutable so you can't simply zero the milliseconds component either.

To get the date only value for use in GP date fields, you may simply use the in build .NET method ".Today" as shown in this snippet;Snippet

DateTime.Today;

 

 Hopefully this will help get back up and running after Googling the problem you've just encountered! - If so do comment, it keeps me motivated to blog more.

Installing Remote Server Administration Tools for Windows 10

Note to self, run following where the add features option in windows doesn't work. 

DISM.exe /Online /add-capability /CapabilityName:Rsat.ActiveDirectory.DS-LDS.Tools~~~~0.0.1.0 /CapabilityName:Rsat.BitLocker.Recovery.Tools~~~~0.0.1.0 /CapabilityName:Rsat.CertificateServices.Tools~~~~0.0.1.0 /CapabilityName:Rsat.DHCP.Tools~~~~0.0.1.0 /CapabilityName:Rsat.Dns.Tools~~~~0.0.1.0 /CapabilityName:Rsat.FailoverCluster.Management.Tools~~~~0.0.1.0 /CapabilityName:Rsat.FileServices.Tools~~~~0.0.1.0 /CapabilityName:Rsat.GroupPolicy.Management.Tools~~~~0.0.1.0 /CapabilityName:Rsat.IPAM.Client.Tools~~~~0.0.1.0 /CapabilityName:Rsat.LLDP.Tools~~~~0.0.1.0 /CapabilityName:Rsat.NetworkController.Tools~~~~0.0.1.0 /CapabilityName:Rsat.NetworkLoadBalancing.Tools~~~~0.0.1.0 /CapabilityName:Rsat.RemoteAccess.Management.Tools~~~~0.0.1.0 /CapabilityName:Rsat.RemoteDesktop.Services.Tools~~~~0.0.1.0 /CapabilityName:Rsat.ServerManager.Tools~~~~0.0.1.0 /CapabilityName:Rsat.Shielded.VM.Tools~~~~0.0.1.0 /CapabilityName:Rsat.StorageReplica.Tools~~~~0.0.1.0 /CapabilityName:Rsat.VolumeActivation.Tools~~~~0.0.1.0 /CapabilityName:Rsat.WSUS.Tools~~~~0.0.1.0 /CapabilityName:Rsat.StorageMigrationService.Management.Tools~~~~0.0.1.0 /CapabilityName:Rsat.SystemInsights.Management.Tools~~~~0.0.1.0



Office Interop Access is denied fix for .NET automation of Microsoft Office

Problem, Could not load file or assembly … or one of its dependencies. Access is denied

This is the solution to a perplexing issue around Office automation, that many people do not seem to be finding a solution for on forums.

The story, on a newly deployed set of PCs, a big issue arose where the Microsoft Office Interop for our .NET application would fail with errors of “Access is denied” when launching our integrating application. Solutions that were attempted proved only to be a temporary fix, with the problem reverting back again, often overnight.

Could not load file or assembly Microsoft.Office.Interop.Excel.dll or one of its dependencies. Access is denied.

Could not load file or assembly Microsoft.Office.Interop.Outlook.dll or one of its dependencies. Access is denied.

It turns out this is to do with permissions to the Interop assemblies that are put in the Global Assembly Cache (GAC) and the Centennial version of the Office App.

 

Solutions attempted

 

In an attempt to resolve the issue, the usual culprits were investigated including many more ideas that I’ve now forgotten about:

  • Changing the permissions within the COM, component services control panel for the access, launch and activation permissions etc
  • Taking control of the assembly cache permissions and granting access for directories and files including…
    "C:\Windows\assembly\GAC_MSIL\Microsoft.Office.Interop.Outlook\15.0.0.0__71e9bce111e9429c\Microsoft.Office.Interop.Outlook.dll"
    "C:\Windows\assembly\GAC_MSIL\Microsoft.Office.Interop.Excel\15.0.0.0__71e9bce111e9429c\Microsoft.Office.Interop.Excel.dll"
    "C:\Windows\assembly\GAC_MSIL\Microsoft.Office.Interop.Word\15.0.0.0__71e9bce111e9429c\Microsoft.Office.Interop.Word.dll"
  • Uninstalling restarting, reinstalling Office
  • Uninstalling restarting, installing the Office interop via the Office installer options
  • Removing and replacing the local Office interop assemblies for the integrating application
  • Embedding the Interops in the application (difficult with Nuget packages tho?)
  • Using cacls.exe to update DCAL

Some of the above would work for the rest of that day, fixing the issue,  yet the problem would return the next morning (machines left on overnight).

 

Root cause

Finally, a hint of the issue was found from digging deeper and deeper in the forums, revealing the root cause.

“Office in the Windows Store” was released which is a different version of Office that is available through the windows store in Windows 10. This version of office is often referred to as Office Centennial version, as it utilises Project Centennial to allow office to be ported to an Office store app. Project Centennial is a bridging solution intended to make it easier for software developers to migrate existing applications to the store and app architecture in windows. This bridge was used to create the version of office in the store, which is distinct from the standard Office you would install from other media sources.

This store version of Office has been included in the Windows system image used by some OEM manufacturers of PCs, so new users have it ready to go on the machine when they get it to the Office or home. Being a store app, it gets refreshed by the auto updates of the Windows store… you can see where this is going?…

It would seem that this version of Centennial Office was installed on the disk image of these PCs and when full Office professional was installed on the machine, all is fine, until the (not in use) Centennial version of Office decides to repair itself overnight during the store update, corrupting the permissions on the Interop assemblies in the GAC, leaving them with only SYSTEM permissions, thus rendering them inaccessible to the end users.

Doing many of the aforementioned attempts to fix the issue, would overlay the correct permissions, but only until the Centennial version of Office went and undid the work overnight.

What is more annoying is that early on in the diagnostics for this issue I had a suspicion around this being the issue, but found no installer under, Start>>Apps and Features for  Microsoft Office Desktop Apps or Office, other than the pro copy. Also nothing in the programs and apps. Hence I’d assumed it can’t be the issue, wrongly as it turns out.

This seems to be an issue that has been hitting people from early 2019 through to 2020 when this blog post was written, although the OEMs were asked not the include the version of Office in the images anymore, so this should fade away over time.

It is a bit concerning that few people in forums seem to have found solutions, other than rebuilding the machine, but pushing on…

 

Removing Centennial Office

It seemed the obvious solution was to remove the store version of Office. It was not required and thus removing it should stop it overwriting the good install of Office. As noted before, there seemed to be no “uninstall” option anywhere in windows to get rid of the application. There were a few ways to remove this Office listed on internet forums, but the only one that worked for me was to launch power shell as an admin user. Then to run the following three commands, each in turn.

"(Get-AppxPackage -Name Microsoft.Office.Desktop).Dependencies | Remove-AppxPackage"
"Get-AppxPackage -Name Microsoft.Office.Desktop | Remove-AppxPackage"
"(Get-AppxPackage -Name Microsoft.Office.Desktop).Dependencies"

The final command should just confirm removal, by not returning anything.

Immediately it could be seen the assemblies had been removed from the “C:\Windows\assembly\GAC_MSIL\*” directories and after running the afflicted application, the issue would seem to be resolved. The professional office install was also repaired, via the Office Pro installer, for good measure.

A very annoying, obscure problem finally resolved!

 

References:

How do I uninstall Office 365 Windows Store version?

how to uninstall office 365 preloaded in windows 10

Why ‘Office in the Windows Store’ isn’t really Microsoft Office

Office.dll access denied to VSTO add-in

SQL indexed views are incompatible with Dynamics GP

Trying to use indexed views with Dynamics GP? – well I’ve been there and I have seen others attempt the same. The Dynamics GP database and application are not really compatible with SQL indexed views. To help out those searching around this subject I thought I should write up our experience.

What is an indexed view?

An indexed view is a SQL database view that has had an index applied to it. This sounds obvious, but the important thing to realise is that the index will be materialised as an index on disk. Think of it as another table created on disk representing the data held in the view, this is why it is quick to get the data if it is summary data. This index "table" is maintained whenever the data that it covers is altered. So it is easy to imagine that indexed views are great for creating summaries, filtered views of data that cross multiple tables, as they will keep track of changes automatically and update the index "table", thus keeping the data "in-sync" with the data in the various tables that it derives from. As the data that represents the SQL view has been pushed to disk and persisted there, querying that summary data is lightweight and quick, because the work has already been done to process the data when it was stored, pre-joined between tables and/or summarised, rather than the database engine having to do all that work on the fly for each query as the query is ran.

However, there is a downside. When (any) index is created, you pay a price for the benefit of fast data recall when reading, by suffering with slightly longer writes times and those longer writes lead to longer living locks on the data and then this can lead to blocking. This may cause issues with performance in some circumstances. The extra writing is because the data held in the index "table" that represents the view needs to be maintained (updated) whenever any of the data underpinning that view changes. Depending on the workload sizes of tables, IO performance etc, this may be significant work if large numbers of records are updated at once. Again as with any SQL index, it takes space on disk too. For large wide views over large tables this may be a consideration too bloating storage and the knock on consequences that brings.

One of the most frustrating matters when working with indexed views is that there are a whole heap of constraints and restrictions around what is permitted in the query that forms them. For example, when using GROUP BY, it must contain the function BIG_COUNT(*) as a column and various database settings restrictions can apply too (there are many, many more). This means when designing the simplest views SQL compile errors warning that it is not possible to do "this and that” frequently cause annoyance. It is very obvious why this is the case when you think about it. It is due to the fact the data is persisted to disk, so it needs to be unchanging to store it, thus any SQL function used by the view has to be deterministic -aha, no “getdate()” function! It is surprising how often this will catch people out. I could go on and on about the restrictions and requirements of indexed views, but just go try making one and you’ll discover the pain yourself, then go read up the documentation to realise how much there is to it!

Another example below are the SQL SET OPTIONS that are required by SQL indexed views…

IndexViewSettings

Using indexed view with Dynamics GP

Professionally we use indexed views a lot in our applications for the speed of access and real time integrity of summary data they offers, so sooner or later a GP admin or GP developer decides they would quite like to use an indexed view... -then it all ends in tears, let us see why…

WITH SCHEMABINDING

This is the first problem. To build an index view the view must use SCHEMABINDING. This locks the database schema and the view together. Unfortunately this may then cause issues with application updates. When Dynamics GP is updated for a service pack or upgrade, sometimes the tables or other database objects may need to be dropped and recreated. This happens during the upgrade process, however if the object that is to dropped is schema bound, then the upgrade script will not be allowed to do what it wants, causing the upgrade to fall over. Hence if indexed views have been bound to the GP tables this is a real risk when a site comes to do an upgrade. Obviously if the person performing the upgrade was aware, then they could drop the view and recreate it after the upgrade, but in real life the knowledge is lost as employees leave or contractors move on causing upgrade pain. 

The second problem is more terminal. Let's try it and see what happens if we create a view over the sales order header and lines to speed up the number of sales lines by country that and not voided in historical transactions. Create the view on a non-production database like this…

 

SET NUMERIC_ROUNDABORT OFF;
SET ANSI_PADDING, ANSI_WARNINGS, CONCAT_NULL_YIELDS_NULL, ARITHABORT,
    QUOTED_IDENTIFIER, ANSI_NULLS ON;
-- We must schema bind
IF OBJECT_ID ('SalesVoidedLinesSummary', 'view') IS NOT NULL
    DROP VIEW SalesVoidedLinesSummary ;
GO
CREATE VIEW SalesVoidedLinesSummary
    WITH SCHEMABINDING
    AS 
       SELECT  COUNT_BIG(*) Cnt,
               SOP30200.SOPTYPE,
               SOP30300.CCODE as Country
       FROM 
       dbo.SOP30200
       JOIN
       dbo.SOP30300 ON SOP30200.SOPTYPE=SOP30300.SOPTYPE AND SOP30200.SOPNUMBE=SOP30300.SOPNUMBE
       WHERE
       SOP30200.VOIDSTTS=0 --Not Voided
       AND SOP30200.SOPTYPE IN (1,2)
       GROUP BY  SOP30200.SOPTYPE, SOP30300.CCODE
GO

 

Then create the indexed view by adding the index. - REMEMBER I SAID NO PRODUCTION DATABASES WITH THIS LITTLE EXPERIMENT!


--We materialise the view by creating an index on it
CREATE UNIQUE CLUSTERED INDEX IX_SOP30200SOP30300VoidSummary
    ON SalesVoidedLinesSummary (SOPTYPE, Country);
GO

 

We can now query the view data like so..

SELECT  * FROM SalesVoidedLinesSummary where SOPTYPE=1 AND Country='PT'

and directly before we created the view with this…

SELECT  COUNT_BIG(*) Cnt,
               SOP30200.SOPTYPE,
               SOP30300.CCODE as Country
       FROM 
       dbo.SOP30200
       JOIN
       dbo.SOP30300 ON SOP30200.SOPTYPE=SOP30300.SOPTYPE AND SOP30200.SOPNUMBE=SOP30300.SOPNUMBE
       WHERE
       SOP30200.VOIDSTTS=0 --Voided
       AND SOP30200.SOPTYPE = 1
       AND SOP30300.CCODE='PT'
       GROUP BY  SOP30200.SOPTYPE, SOP30300.CCODE

 

...and we find that from not having the view to having the view we have dramatically reduced the query time as shown below. I’m not sure about the integrity of those figures but this isn’t a discussion around query optimisations, just accept that indexed views solve performance problems.

    Total execution time without view:  17390   
     Total execution time with view:        505   

 

In production

So developer or IT pro after testing that out on the test company database says, “great, now lets create it on the production database as that worked like a charm”, and then goes to login to GP and create a sales order having created the view…

...to then get the dismay of the error that follows...

“An edit operation on table ‘SOP_Master_Number_SETP’ failed. A record was already locked'.”

IndexViewGPError1

and

INSERT failed because the following SET options have incorrect settings ‘QUOTED_IDENTIFIER, CONCAT_NULL_YEALS_NULL,ANSI_WARNINGS’. Verify that SET options are correct for use with indexed views and”…

GPIndexViewError2

… then starts the support calls flowing in from the users! 

So why we can’t have nice things?

Go back up in this post, as you saw in the graphic, that there are certain SQL SET options required to make indexed views work. Sadly these are not compatible with the SET options required to make Dynamics GP work shown below! -thus the application breaks, simple as that! If those phones are still ringing from your users right now, then just simply drop the view and your users will be happy again.

We did play around a bit but never found a solution to this issue, but if you have a work around do let me know.

References

Mariano Gomez has a nice post about why this is required by Dexterity to make GP run correctly in this post:

Microsoft SQL Server DSN Configuration

Microsoft - Creating Indexed Views

Gavin wrote this which inspired me to document the issue before others fall into it...

Brief overview and comparison of how summary values are stored and calculated in Dynamics GP, Dynamics NAV and Dynamics 365 Business Central