Replicating GP price table to website
To provide our website with bang up to date product prices as they are in our ERP system, we replicate the price table from our ERP system to the website SQL server database. The price table holds nearly two million price rows consisting of many combinations of currency, item, price quantity, units of measure, discount breaks and customer specific price lists.
The replicated table works great, until a big price update is required. If most of the prices are updated say in line with inflation, it hits a good number of those rows in the database. This causes a BIG transaction to make its way through the relatively thin wire to our website. From opening the transaction (and thus locking the subscriber table) to committing the transaction can take a long time locally and then for that to make its way through the slow connection to the website and be committed. All these processes take a finite amount of time. The lock caused on the price table at the website database while all this is happening causes a problem. That lock caused any reads on the price table to be blocked for a long time until everything had passed through, bringing the website to a halt for tens of minutes.
To avoid this queries at the website that interrogate the replicated publisher table could be set to READ_UNCOMMITTED transactions. However, potentially this could lead to problems in reading “dirty records” that are not ready for public consumption. This is significant when you consider these are price tables and reading prices that are in an unknown state is a no-no.
One was to take a database snapshot before the bulk price update, switch all the views on the table to use the snapshot letting replication take its time and update the records in the underlying table. Once finished the views could be switched back to point at the original table again. This could, perhaps be controlled by a replicated signalling table from the ERP system so that we don’t have to issue SQL between the databases. It should work well in that once the all clear signal is set in the publication database, it will not propagate through to the subscriber until after all the other changes in the log have been replayed and committed to the subscriber.
The second idea was to switch the subscriber database into ALLOW_SNAPSHOT_ISOLATION ON mode. Simply by executing the following commands:
ALTER DATABASE [ERPReplicationdatabase]
SET ALLOW_SNAPSHOT_ISOLATION ON
ALTER DATABASE [ERPReplicationdatabase]
SET READ_COMMITTED_SNAPSHOT ON
Once the database is in snapshot isolation mode, the reads will use versioned row numbers from the database. The reads are never blocked as when a transaction begins, the row version before the transaction and below is used for any reads. So reading the website is not blocked while a transaction is underway, and what is read is the state before the transaction started, that for our application is perfect.
No DML commands need issuing or signalling between the databases. This is the cleanest solution for what is required. The next task is to ensure that all the changes made to the price tables are made inside one transaction to keep the reads off any of the new changed data until is fully committed to the database.
After pulling a very large solution into smaller projects over the last da or two, a windows forms event handler that used to work, started complaining about the signature not matching for the event. A custom class inherited from EventArgs and added some properties for this event. The definition for the custom event argument resided in one of the class libraries that had been refactored.
To the eye and the intellisense tooltips the signatures looked identical. The break through was found by right clicking on one of the event arguments to “find definition” that lead to the class viewer rather than the code defining the class. This was a clue as this normally only happens when the source is not loaded in the IDE or the project is not holding a project reference, rather a file reference to the .dll of the class library in question.
Furthermore clicking the “find definition” on the other side of the event handler did lead to the source definition for the custom class.
I looked in the class viewer in visual studio and found two identical copies of the class in there. Somehow a cached version of the old .dll pre-refactor was still in the solution. After deleting the /obj and /bin contents for all projects and taking the references out and back in again, it all came back to life as it should.
“String or binary data would be truncated”, an error raised by SQL server when trying to update or insert data into a column that is bigger than that column can accommodate.
On my wish list for SQL server for a long time has been that this error should provide the column that raised the error as part of the reported error. Our Microsoft ERP system has very wide tables, approximately two hundred columns in some of them. When this error happens, it becomes a pain to track down the failing column update.
The only way I’ve got to do this at the moment is to add the LEN() function around every possible culprit and check the returned lengths manually against the schema. Whilst this works, it takes a long time, just for lack of the column name in the error raised.
If you need to connect to TFS (Team Foundation Server) from Visual Studio 2008 remember the collection. Use a connection similar to this…