Computer%20with%20Graph_edited.jpg

SkyBITS Release Version 1.3

The SkyBITS team formally released SkyBITS v1.3 on Friday, April 23, 2021. There are many new exciting features across the application including features to the Data Modeler, Orchestration, and other productivity and stability enhancements. Plus, this is the first release under our new name: Bion. We will be changing the brand over from SkyBITS to Bion in the coming weeks. The list of features and enhancements is included below:

 

Orchestration

  • Synapse dedicated and serverless endpoint connection types have been added for use by the General Data Mover Task.

  • When a batch is paused or is in an error state, any tasks that have not yet run may now be edited. Previously, the batch had to be cancelled first.

  • Batches can now be "killed." This allows a batch to be stopped without the need to wait for running tasks to complete.

  • Batches can now be canceled while running, without requiring the user to select "Pause" first. Behind the scenes, it transitions to paused, then cancelled automatically.

  • Certain schedule types now allow the exclusion of specific days and can be configured to run only within certain windows of time. Example, a batch can now be scheduled to run every 5 minutes, but only between the hours of 12 and 3 PM, and not on Sundays.

 

Data Modeler

  • When deploying a data model, the server and database are now registered in the Bion database and are visible in the User Interface. Historical deployments can now be reviewed here.

  • Data Model tables can now be created by a user created query. (Bion will inspect the query results and create a data model table from it).

  • For new Extracts, the behavior of how missing _dataflow records are handled is more intuitive. For SQL extracts, a variable has been added that will raise an error if missing instead of silently failing. The extracts will now run if the record is missing (in cases where it is not used). For Data Movement extracts, the task will error out if the variable is referenced, otherwise will ignore the fact that is is missing (and unneeded).

  • Added support for pre and post deployment scripts during Data Model deployment.

  • Added support for multiple versions of ETL Templates. This will allow smooth upgrades to Bion versions with ETL/DW improvements without forcing customers to take the changes if they desire not to.

  • Added a screen which allows the viewing and customization of the templates used to generate Data Warehouse objects. While generally it is not recommended that the existing templates be changed (as this may cause upgrade conflicts), it is sometimes necessary. A screen is now available to allow this. The feature must be turned on in the user settings. (US 100713) - Modifying templates may result in conflicts when new versions are installed. This feature allows template conflicts to be resolved within the UI. 

Other Enhancements

  • When deploying a data model to a new Azure database and the database does not exist, Bion now defaults the new database to the "Basic" tier. This database should be scaled up as needed.

  • Some data model table defaults have been changed to more typical values including: Deployment Archive Copies = 0, Skip Build of DQ Checked, and CheckFor Deletes Unchecked. Also new columns are now of a consistent type (no longer a mix of varchar and nvarchar).

  • Two hidden JSON parameters have been added to model json files which control the update statistics behavior present in transform procedures. The first, SkipStatisticsUpdate, indicates if the stats update should be skipped all-together. The second, StatisticsUpdateMethod, contains the text to use if it is active. For example, "SAMPLE 20 PERCENT". This setting is present for each table. The model must be exported, edited, and then imported again, then the model must be redeployed for this setting to take effect.

  • Bion now sets Read Committed Isolation level off due to inconsistent SQL Server behavior when the setting is enabled. This setting should be off for proper Bion functionality.

  • Scheduled batches will no longer run if the batch has already been running or was in a paused or error state when the new scheduled time elapsed. This avoids confusion when a batch finishes or is cancelled and runs unexpectedly.
     

All services must be upgraded for this major release.