Feeds:
Posts
Comments

This is basically the MSDN sample with the addition of an user-supplied separator. For some reason I couldn’t get this to deploy successfully from within visual studio I had to create aggregate manually inside SQL Server (2008).

The code for the aggregate

using System;
using System.Data;
using Microsoft.SqlServer.Server;
using System.Data.SqlTypes;
using System.IO;
using System.Text;

[Serializable]
[SqlUserDefinedAggregate(
 Format.UserDefined, //use custom serialization to serialize the intermediate result
 IsInvariantToNulls = true, //optimizer property
 IsInvariantToDuplicates = false, //optimizer property
 IsInvariantToOrder = false, //optimizer property
 MaxByteSize = -1) //maximum size in bytes of persisted value
]
public class Concatenate : IBinarySerialize
{
 /// <summary>
 /// The variable that holds the intermediate result of the concatenation
 /// </summary>
 private StringBuilder intermediateResult;
 /// <summary>
 /// Initialize the internal data structures
 /// </summary>
 public void Init()
 {
 this.intermediateResult = new StringBuilder();
 }
 /// <summary>
 /// Accumulate the next value, not if the value is null
 /// </summary>
 /// <param name="value"></param>
 /// <param name="separator"></param>
 public void Accumulate(SqlString value, SqlString separator)
 {
 if (value.IsNull)
 {
 return;
 }

 this.intermediateResult.Append(value.Value).Append(separator);
 }

 /// <summary>
 /// Merge the partially computed aggregate with this aggregate.
 /// </summary>
 /// <param name="other"></param>
 public void Merge(Concatenate other)
 {
 this.intermediateResult.Append(other.intermediateResult);
 }

 /// <summary>
 /// Called at the end of aggregation, to return the results of the aggregation.
 /// </summary>
 /// <returns></returns>
 public SqlString Terminate()
 {
 string output = string.Empty;
 //delete the trailing comma, if any
 if (this.intermediateResult != null
 && this.intermediateResult.Length > 0)
 {
 output = this.intermediateResult.ToString(0, this.intermediateResult.Length - 1);
 }

 return new SqlString(output);
 }

 public void Read(BinaryReader r)
 {
 intermediateResult = new StringBuilder(r.ReadString());
 }

 public void Write(BinaryWriter w)
 {
 w.Write(this.intermediateResult.ToString());
 }
}

The  SQL to create the assembly and aggregate

create assembly SqlClassLibrary
 from N'C:\temp\SqlClassLibrary.dll'
 with permission_set = safe
go

create aggregate concatenate (
 @value nvarchar(max),
 @separator nvarchar(10)
) returns nvarchar(max)
external name SqlClassLibrary.Concatenate

go

Sample usage

select module_name, table_name,
 concatenate(column_name, ', ')
from load_file_mapping
group by module_name,table_name

- a large glass
- about half a handlfull of mint
- 1 tablespoon sugar
- crush with motar for a few seconds
- fill with crished ice (largish bits)
- add 2 shots Captain Morgan Spiced Gold
- fill with ginger ale

awesome summer drink…

I was recently tasked with creating a SQL Server Integration Services (SSIS) package to load a flat file that contained records of varying lengths destined for separate tables.  Having just come out of a 16 month project where I was using Pentaho’s PDI toolset for ETL I had happy thoughts thinking I would I have this wrapped up in time for arvo beers.

I was sorely disappointed thought as what should have been a simple task turned out to be laborious mess.  In all fairness to SSIS there may be a better way to do this and if there is please let me know as I failed to find anything.  So back to the statement of work, load a flat file containing records of various lengths into a staging table that has as many columns as the longest record.

Using Pentaho this is a simple pleasant experience. It requires  a transformation with a “CSV file input” step that loads the file which is tab delimited, and table output step that gets the records and loads them into a table . There ya’ go 2 minutes later and I’m done, hell it even scanned the file and suggested datatype’s and generated SQL for the destination table… awesome.

pdi transformation

PDI Transformation

Using SSIS 2008 it turns out it’s not that nice. SSIS has a “Flat File  Source” step that you use to pull in data from flat file types. The only caveat is it doesn’t support files that have different record lengths. At first I thought I could get past this as it was just scanning the first row to get the number of columns, but it turns out it the just carries on eating chars, including the specified record delimiter char(s), until it has enough values for how many columns the first record had, and then just stuffs in leftovers in the last column. What this means is that for records that are “short” it starts eating into the next records values. (I was however pleased to see that Microsoft’s Data Transformations Services (DTS), yes that previous generation product by a few years, handles this scenario perfectly). So how do you this in SSIS well you have read the data into one large string column, then slap on a “Script Component” and define output columns and their data types one-by-one  (I had 220 output cols yay). Then you write a script to split the rows and populate values for all you output columns, and finally a “SQL Server destination” step to load the data.

Needless to say I’m doing my bit to get Penthao PDI in here.

Always handy from PostgreSQL Online Journal

How to determine the size of a database on disk

SELECT pg_size_pretty(pg_database_size('somedatabase')) As fulldbsize;

How to determine the size of a database table on disk

NOTE: There are two functions in PostgreSQL – pg_relation_size and pg_total_relation_size. The pg_relation_size just measures the size of the actual table where as the pg_total_relation_size includes both the table and all its toasted tables and indexes.

SELECT pg_size_pretty(pg_total_relation_size('someschema.sometable')) As fulltblsize, pg_size_pretty(pg_relation_size('someschema.sometable')) As justthetblsize;

How to determine the size of a database schema

When you are doing a lot of data loading for later massaging to dump into your production tables, you will often generate junk tables in the process and worse yet other people may generate junk tables period. As a general practice, we like to create a schema just for junk. The reason for that is that it is extremely easy to exclude schemas from being backed up and to load individual schemas. So with this approach we create or move all tables we are not quite sure if they are needed or we only need temporarily into a schema called junk, scratch, possiblejunk or whatever you want to call the schema and exclude it from our backups.

Something of the form:

CREATE SCHEMA scratch;
ALTER TABLE someschema.joeyplaying SET SCHEMA scratch;

After saving the world from junk, you would like to know how much space you have saved your backups from backing up. So you would employ a query something of the form:


SELECT pg_size_pretty(pg_database_size('mycurrentdb')) As fullprod,
pg_size_pretty(CAST(pg_database_size('mycurrentdb') - (SELECT SUM(pg_total_relation_size(table_schema || '.' || table_name)  )
FROM information_schema.tables WHERE table_schema = 'scratch') As bigint)) As  tobebackedup_size,
pg_size_pretty(CAST((SELECT SUM(pg_total_relation_size(table_schema || '.' || table_name) )
FROM information_schema.tables
	WHERE table_schema = 'scratch') As bigint) )  As junk_size;



I’m busy wrapping up my first major solo project, Sanxploit, a Storage Resource Management (SRM) application of sorts that sits on top of EMC Clariion arrays and provides business and technical reporting. This is a relatively large application and capitalizes on a few good open source projects namely Rails, PostgreSQL, JasperServer and Pentaho PDI. The project is now moving into a beta sales phase.

Some screenshots

Dashboard view gives you a quick overview of your storage

Dashboard view gives you a quick overview of your storage

Users can categorize Storage Group's for aggregated reporting

Users can categorize Storage Group's for aggregated reporting

Headroom report shows allocated vs. free storage per drive-type

Headroom report shows allocated vs. free storage per drive-type

Storage Allocation report shows storage by category

Storage Allocation report shows storage by category

I’m busy trying to create my first ruby library, a jasperserver web services client, after not finding anything suitable out there. Since I’m not that strong on the ruby front I’ve started by “copying” another library.  What I have realized, and it something that exists allot in other environments like microsoft dev, is the proliferation of bad code out there.  So make sure your “learning” project is based on something decent else you gonna be bad and produce more bad when somebody repeats what you’ve done.

Do you often need to manipulate controls in prompt pages in cognos? Well if you do there is one major point to take in consideration – cognos intrinsic control names. To access any of the cognos prompt controls via javascript you will need to supply the cognos intrinsic control name plus the given name in order to identify the control.

Now imagine its two years down the line and your busy upgrading to a new version of cognos. You run a report and bam, javascript errors abound on your custom prompt page. No worries you fire up firefox/firebug and low and behold cognos have decided to rename their intrinsic controls. You start sweating trying to calculate how may hours its gonna take to go through two years worth of reports to update your javascript.

Well here’s a very easy solution to abstract away this dependency. Follow these steps (Cognos 8+ & Windows, requires admin access on cognos server(s) and a system bounce):

1. Navigate to <congos-install-dir>\webcontent

2. Create a new folder, I will use etc for the this example

3. Create a js file in your newly created dir, mine is <congos-install-dir>\webcontent\etc\etc.js

4. Add some javascript to hold the cognos control names e.g.
var Cognos = {
Dropdown: '_oLstChoices',
Textbox: '_textEditBox'
}

5. Now open the file <congos-install-dir>\templates\ps\prompting\p_include.xsl

6. Search for the text
<xsl:if test="//promptButton or //defaultPromptFooter or //selectValue or //textBox or //selectDate or //selectWithSearch or //selectWithTree or //selectTime or //selectInterval or //selectDateTime or //selectDataSourceSignon">

7. Add a script element below this to include the js file – mine is

<script language="javascript" src="../etc/etc.js" type="text/JavaScript">//leave this comment here to assure we have an explicit end tag for script</script>

8. Save and close, bounce cognos

Now all cognos prompt pages will include this js file, so if cognos one day decides to rename their controls you can modify the names in the js file and your set. Obviously you can take this one step further and create a library providing macros for routine prompt screen javascript stuff.

technorati tags:,

Blogged with Flock

For a long time I have been using the cognos forums on ittoolbox.com, but then the other day someone put me onto cognoise.com and well I was pleasantly suprised. Do yourself a favour and check them out they definitely going places.

Older Posts »