SQL to extract contiguous ranges for maintenance tasks in Dynamics GP

From time to time I find some SQL that really makes me smile and sit back and stare at it with awe. Today is one of those days, let me show you…

Many of the maintenance windows in Dynamics GP ask for ranges of numbers to be entered. They consist of a start number and then an end number, start and end points for which you would like to execute a operation over.

It is a common thing in GP is to use a GP macro mail merge to automate the user interface where a repetitive operations are required.  If a macro mail merge is being used for one of these windows, then it requires the start and end numbers over which we would like to perform the operation.

Let move to something more tangible and the real world example I’m working on. However don’t get tied down by my example, there are many other range windows in GP that this principle would work on too.

Remove Sales History Windows Dynamics GP

I want to surgically remove 901,872 historical sales documents from GP, using a macro. I am using a macro so that we are safe, the GP biz logic gets applied by the UI. I could use SQL but always best to try macro first so to benefit from the safety the UI gives us. This could take some time, supplying the document numbers and one by one deleting them.

Master numbers are a number id that ties together a string of sales documents that are related, it is held against all sales documents. For efficiency we notice that we can remove documents by master number, so all related documents, quote->order->invoice->return will be deleted at the same time. This would mean looping through the remove operation of window much less, as one master number removes many documents.

This brings us to 446,741 unique master numbers that need removing from 901,872 sales documents (yes I’ve checked and only three master numbers in my range link to related documents outside my range, so I’ve taken those out of this list). I could macro the window to set the "from" and "to" to be the same master number, thus removing one set of related documents at a time, then moving to the next number and then the next. It would work, but testing showed I’d be here all week doing it. Besides I can see somehow that I can be more clever than that!

GP is more efficient on these types of operation if you can supply ranges of numbers to work on, so really I want to look at the list of 446,741 numbers and find all the contiguous ranges within the sequence. Then identifying the start and end numbers to put in the UI start and end fields. Although it is worth remembering sometimes there will just be on number in a particular range, when there are gaps on both sides of the master number.

Hitting Google I found the following post, where they use recursive CTEs to find the ranges. Restricting my sequence to only 10,000 records, it took about three mins to run. Letting it run on the full sequence, I ran out of patience (and probably server resources) to let it get to end. I did try to tune it with some physical tables indexed to try and improve performance, but ultimately recursive CTE’s are terrible for large data sets like this.

How to find contiguous ranges with SQL

I went looking again and found another solution in this post.

How to find the boundaries of groups of contiguous sequential numbers?

This is genius insight in its script design. I do use ROW_NUMBER() a lot myself for all sorts of weird problem solving, but had never seen it applied like this before.

I adapted the snippet of the SQL to my problem and came up with this:


FROM [dbo].[SOP30200] --WHERE Blah blah...

AS (
) - MSTRNUMB AS Groups
So from a query that never finished processing we go to this:

query times

Really, it is now down to a few seconds, to get all the results!


We have now only 3,760 row ranges, from 446,741 master numbers, so I now only need to get the macro to iterate the remove window 3,760 times, that is a much more acceptable!

So the mail merge macro now takes each row of this result set and uses the from and to values from the query to populate the boxes for the master number and then sets the report options and finally then processes. Job done.

Back to the SQL for a moment…

The SQL is fascinating for this though, looking at the CTE data we see how it works.

sql result cte

The row number increments but the subtracting of the ROW_NUMBER from the master number sequence causes a banding or grouping number to be created. So long as the sequence number is only one greater than the previous number, then the banding number is maintained the same in the group. This is because the relative difference between the ROW_NUMBER and the master number has not changed. If the master number jumps up because of a gap it will have gone up by more than the row number, so the grouping will number will be changed to a new grouping number.

This grouping number is then subsequently used to gather the max and min values for each section. I love the efficiency and simplicity of this solution. It also shows how many ways there can be to solve a problem.


Further improvements

I can see further improvements are possible. If we in-fill the sequence of master numbers before processing them with dummy master numbers, where documents don’t exist in the database, then this will further reduce the number of times we have to loop. In that case it would allow cases where the max of one range to the min of the next contain no documents to be combined into one range.

I’ve ran out of time and leave it to the reader to investigate that optimisation.

Error: This document contains one or more posting holds, Dynamics GP Edit List

If you are experiencing this on SOP Sales Invoice posting edit lists, then maybe you are using drop ship items. Also perhaps you have the Shipping Notification Tool installed?

Error this document contains one or more posting holds

There is no posting hold as such on the document, the posting hold mechanism is being hijacked by the shipping tool to prevent posting of the document. It is actually because GP does not think the item in question have been shipped yet and so is preventing it from posting. Ensure that the item was marked as shipped using the ship notification window.

The setting in the INI file needs changing if you want to change this behaviour, details of the shipping notification window are in this post as is more information on the INI settings to change if you want to change the behaviour:

 Dynamics GP Drop Shipping Sales invoices before purchase invoice with Shipment Notification Tool (SNT)

Empty Matched To Shipment in Purchasing Invoice Entry of Dynamics GP

About from time to time on Euro transactions we get a yellow triangle and empty “Matched to Shipment” field in the Purchasing Invoice Entry window of GP.

Purchasing Invoice Entry Yellow Triangle

The problem is always the same, the fields


Are all either default or missing for the item row in the table POP10500.

I am still looking for the break through clue as to what causes this, I’m guessing network outages or other hardware failures interrupting processing somewhere. In the meantime I just fix the issue when it comes up, but having fixed it a couple of times I wrote a script to help.

If this is the problem that you are seeing too then the following script can be used to fix it for a given PO Number.


FROM POP10100 ph
WHERE ph.PONUMBER = '{your po number}'


As always, I can’t know your GP environment, so this should be thoroughly tested in a test environment before running the script on production data. I can’t take any responsibility for what might happen on your systems.

Let me know if this helped you in the comments, it motivates me to blog more…

OLE Notes Migration

It was time to get our old OLE notes migrated. GP has stopped using OLE notes containers for attachments to notes in the system, mostly due to the need to work with the web browser hosting of GP.  I didn’t want to import the attachments back into the new GP attachments feature, as no one had been screaming for the ones that had vanished after the upgrade (only a few enquiries). However I did want to make them available should someone need them enough (don’t ask me for criteria for the enough!). So the plan is to run the first part of the migration tool only, the extract and not the import. For details of the tool see the post by encore:

The OLE note migration utility in Dynamics GP

The OLE notes directory totalled 46GB with 30k documents, running the migration utility brought the extracted size down to 3GB. Evidence is that the object containers were not efficient!

So the lesson is that you might want to extract your OLE notes even if you don’t intend to use them as it will help your file sizes, in our case the storage is backed by a SAN that will be compressing things anyway, extracting the files makes them useable from the directory.

What you get after extraction is a directory structure that starts with the dynamics GP product ID, and then has a folder for each noteidx under that. The note index is the record id for the note in the notes table. So for every product that uses notes and had attachments there should be a top level folder, then subfolders for the note attachments.

It looks like the same kind of scheme used by the newer GP attachments feature, see the post:

Document attach feature Dynamics GP database and BusObjKey formats

the directory name is the hex version of the note index that the contents of that directory link to.

This is good enough for me to recover any documents requested by looking at the note index and converting it.