Announcing Log::Contextual

Author: fREW Schmidt

22 Feb

I really should have posted this sooner. Certainly before I began my next project. Oh well.

I am proud to announce the next bit of mstware! Log::Contextual is a small module for making your life easier when it comes to logging. Instead of bringing yet another logging infrastructure into the mix (see Log::Log4perl and Log::Dispatch), this module is a thin wrapper around any logging system you choose to use. (Note: we are working with authors of major logging packages to work seamlessly with L::C, but at the time of writing most need some form of adapter.)

There are a few major features worth noting. First off, ridiculously convenient interface. Once you’ve set up your logger (presumably in the startup of your app or whatever) all your logging code will look like the following:

1
2
3
4
5

use Log::Contextual qw( :log );
sub hello_world {
log_trace { 'entered hello world' };
# ...
}

Another great thing is that, like Devel::Dwarn, all of the logging functions are identity functions; that is, they return their arguments. That means you can do cool things like the following:

1
2
3
4
5
6

use Log::Contextual qw( :log );
sub hello_world {
my ($arg1, $arg2) = log_trace { "entered hello world with args $_[0], $_[1]" } @_;

# ...
}

Of course, in Perl you may be passing around complex references and the above will get cumbersome fast, so we added shortcuts specifically for logging out data structures:

1
2
3
4
5
6

use Log::Contextual qw( :dlog );
sub hello_world {
my ($arg1, $arg2) = Dlog_trace { "entered hello world with args $_" } @_;

# ...
}

The automatic stringification is done with Data::Dumper::Concise, so you will get reasonably indented output for free.

In a separate package we are going to provide a module to basically turn the logging functions into no-ops at compile time, thus giving you the ability to have your code run just as fast if it never had the logging functions in in the first place. I’ll post more on that once it’s released.

So what are you waiting for? Go log stuff!

0 Comments

Filed under: Uncategorized

New DBIx::Class::Journal

Author: fREW Schmidt

18 Feb

We are planning on using DBIx::Class::Journal in a couple different projects of ours. Unfortunately DBICJ has foundered for more than a year as a dev release, and even that won’t test correctly on win32. So I’ve done some work on it to get it in a more releasable state. I just released a developer release to CPAN and if you currently use DBICJ please test it out. Even if you don’t use it at least test it so I can see if I get failures. The release should end up here, but it will take a couple hours.

1 Comment

Filed under: Uncategorized

New stuff in DBIx::Class::Helpers 2.00200

Author: fREW Schmidt

7 Feb

A new release of the resplendent Perl ORM DBIx::Class means new release of DBIx::Class::Helpers

The ResultSet::Random helper had the wrong function used for MySQL. That was fixed thanks to an RT from pldoh.

get_namespace_parts from the util package was unnecessarily strict. Thanks to melo for the prodding to do that.

I refactored some of the code in core DBIx::Class so that I can more easily detect is_numeric with Row::NumifyGet, instead of requiring the user to specify it. Normally DBIx::Class autodetects it based on column type, but that code wasn’t quite generic enough until now. Nice!

And then the most exciting bit is a new helper entirely for the suite: Row::ToJSON. Basically I was sick of doing this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

package ACD::Schema::Result::Foo;

# regular package stuff here

sub TO_JSON {
my $self = shift;
return {
id => $self->id,
foo => $self->foo,
# etc etc ad nausium
}
}

"distraction";

Of course that can be shortened to:

1
2
3
4

sub TO_JSON {
my $self = shift;
return { map +( $_ => $self->$_), qw{id foo ...} }
}

But I still have to make that stupid columns list! This shiny new helper makes a TO_JSON method that will simply include all of your columns except for the “heavy” ones like TEXT, NTEXT, or BLOB. Of course you can have finer-grained control than that by explicitly saying to include (or not) a column in it’s configuration. See the docs for all the nitty gritty details.

0 Comments

Filed under: Uncategorized

An Exposition on Specific Time Saving Code

Author: fREW Schmidt

29 Jan

I write a lot of ExtJS grids at work. I have written JavaScript classes for our Ext grids that generate as much as possible automatically, but the actual column definitions of the grids are almost always unique. The project I am on now is nearing our first real deploy, and we’re late, so things have been really, really busy.

It wasn’t until recently that I realized just how much time I spent working on grids and their related records (representation of the rows of a grid.) Although even if I’d known just how much I do this at the beginning of the project, I certainly didn’t know DBIx::Class as much as I do now, in addition to the other 4 non-core modules that I’ll mention.

Because I’ve been working all these super long hours (12-14 a day), often after a frustrating day I’ll try to code something more relaxing and rewarding at home or at work but not during work. So today I decided to finally take the plunge and do what I’ve been pondering for a few months and write some code to generate Ext scaffolding for me. So I’m going to walk you through the script that I wrote (and will probably work on more as time goes by.)

This isn’t a full module, not even in my repo yet, so it’s all just in one file. Here’s the boilerplate intro:

1
2
3
4
5
6
7
8
9
10
11
12

#!perl

use strict;
use warnings;

use feature ':5.10';

use Syntax::Keyword::Gather;
use String::CamelCase ();
use Lingua::EN::Inflect ();
use List::Util;
use Statistics::Basic 'mean';

That’s pretty basic. We use a bunch of modules that I’ve used in at least 1 other project before and was happy with.

1
2
3
4
5
6
7
8

use FindBin;
use lib "$FindBin::Bin/../local/lib", "$FindBin::Bin/../lib";
use ACD::Schema;
use Config::JFDI;
my $config = Config::JFDI->new(name => 'acd', path => "$FindBin::Bin/../acd");
my $config_hash = $config->get;
my $connect_info = $config_hash->{Model}{DB}{connect_info};
my $schema = ACD::Schema->connect($connect_info);

That’s the code to parse any kind of Catalyst config file and the grab a new schema based on it. It’s a ton of biolerplate but I live with it.

So next up is a basic inflection function that gives us all the different forms of a word we might need. It starts with either “single_foo” or “SingleBar” and gives us six variations based on that:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

sub inflect {
my $word = shift;

my $return = { singular => {}, plural => {}};
if ( defined $word->{camel}) {
$word = $word->{camel};
$return->{singular}{camel} = $word;
$return->{singular}{noncamel} = String::CamelCase::decamelize($word);
} else {
$word = $word->{noncamel};
$return->{singular}{camel} = String::CamelCase::camelize($word),
$return->{singular}{noncamel} = $word;
}
$return->{singular}{human} = join(
q{ }, map ucfirst $_,
split /_/, $return->{singular}{noncamel}
);

$return->{plural}{noncamel} = join( q{_}, split / /,
Lingua::EN::Inflect::PL(
join q{ },
split /_/,
$return->{singular}{noncamel}
)
);
$return->{plural}{camel} = String::CamelCase::camelize(
$return->{plural}{noncamel}
);
$return->{plural}{human} = join( q{ }, map ucfirst $_, split /_/,
$return->{plural}{noncamel}
);
return $return
}

So that uses String::CamelCase and Lingua::EN::Inflect combined with join and split mostly.

We’re *almost* ready to generate a record. But first we need to define a mapping from the data type in the database to the data type that Ext uses:

1
2
3
4
5
6
7
8

my $types_xform = {
int => 'int',
float => 'float',
varchar => 'string',
bit => 'boolean',
datetime => 'date',
money => 'float',
};

Ok, now let’s look at the code to generate an Ext.record:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

sub generate_record {
my $schema = shift;
my $source_name = shift;

my $source = $schema->source($source_name);

return qq{Ext.ns('ACDRI.record');

ACDRI.record.$source_name = Ext.data.Record.create([\n} .
join(qq{,\n}, sort { $a cmp $b } gather {
for (map [$_, $source->column_info($_)], $source->columns) {
my ($column, $info) = @{$_};
take " {name: '$column', type: '$types_xform->{$info->{data_type}}'}";
}
}) . "\n]);";

}

So what’s going on here is that we get the source from the schema. The source could be considered something like a table definition, although it can also point at a view or whatever too. Then we start generating the string representing our record, and then we use a join/gather combo to get the column data the way we want it.

We could certainly just use a more complex map instead of the for+gather that we have, but I personally feel than any map where you *must* use the block form is cumbersome. So we join together all of the strings that gather took, and then append the end of the definition, and voilà, we have a record!

Here’s example output on our Customer source:

1
2
3
4
5
6
7
8
9
10
11

Ext.ns('ACDRI.record');

ACDRI.record.Customer = Ext.data.Record.create([
{name: 'color_code', type: 'string'},
{name: 'comments', type: 'string'},
{name: 'id', type: 'string'},
{name: 'last_edited_date', type: 'date'},
{name: 'name', type: 'string'},
{name: 'price_approval_required', type: 'boolean'},
{name: 'sales_representative_id', type: 'int'}
]);

If you’re still following along you should be fine with the next, much more impressive functionality.

I’d like to have the script do as much as possible to speed up my work, so let’s see how far we can take this. First, I made a mapping from type to renderer, so that things that are datetimes use our custom datetime renderer, same with booleans:

1
2
3
4

my $renderer_from_type = {
datetime => 'ACDRI.fn.Renderers.dateTime',
bit => 'ACDRI.fn.Renderers.bool',
};

Next, I wrote a function that would try it’s best to guess how wide a column should be based on it’s header and the average width of the strings inside of it:

1
2
3
4
5
6
7

sub avg_width {
my ($rs, $col, $header) = @_;
List::Util::max (
mean(map +( defined($_) ? length "$_" : 0 ), $rs->get_column($col)->all), # average width of field
length $header # col header
);
}

Now finally we have the grid function:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77

sub generate_grid {
my $schema = shift;
my $source_name = shift;

my $source_names = inflect({ camel => $source_name });
my $source = $schema->source($source_name);

return qq`/*global Ext */
/*global ACDRI */
/*global MTSI */
Ext.ns('ACDRI.ui.grid');

/**
* \@class ACDRI.ui.grid.$source_names->{plural}{camel}
* \@extends MTSI.ui.Grid
*/
ACDRI.ui.grid.$source_names->{plural}{camel} = Ext.extend(MTSI.ui.Grid, {
title: '$source_names->{plural}{human}',
record: ACDRI.record.$source_names->{singular}{camel},
updateConfig: {
xtype: 'ACDRI.ui.form.$source_names->{singular}{camel}'
},
addConfig: {
xtype: 'ACDRI.ui.form.$source_names->{singular}{camel}',
},
initComponent: function () {
//this.sortInfo = {
// field: 'part_id',
// direction: 'asc'
//};
var config = {
url: '$source_names->{plural}{noncamel}',
itemName: '$source_names->{singular}{human}',
columns: [` .
join (qq{, }, gather {
for (map [$_, $source->column_info($_)], $source->columns) {
my ($column, $info) = @{$_};
my $colnames = inflect({ noncamel => $column });
my $renderer = $renderer_from_type->{$info->{data_type}}
? "\n renderer: $renderer_from_type->{$info->{data_type}},"
: '';
my $width;
given ($info->{data_type}) {
when ('varchar') {
$width = int 7*avg_width(
$source->resultset, $column, $colnames->{singular}{human}
);
}
when ('datetime') {
$width = List::Util::max(
int 7 * length $colnames->{singular}{human},
57
);
}
default {
$width = int 7 * length $colnames->{singular}{human};
}
}
take qq[{
header: '$colnames->{singular}{human}',
dataIndex: '$column',
sortable: true,
hidden: false,$renderer
width: $width
}];
}
}) . qq`]
};

Ext.apply(this, Ext.apply(this.initialConfig, config));
ACDRI.ui.grid.$source_names->{plural}{camel}.superclass.initComponent.apply(this, arguments);
}
});

Ext.reg('ACDRI.ui.grid.$source_names->{plural}{camel}', ACDRI.ui.grid.$source_names->{plural}{camel});
`;
}

We see nearly the same structure here that we did with the record, except much more intricate. Notice the code to generate the width uses the magic constant 7. As I played with this I found that 7 seemed to work for the font that Ext uses by default (Arial?) Optimally I would actually use some kind of metrics package to ask it for the width of all of the strings that I generated in the function above and average that, instead of averaging character lengths. But this seems to work really, really well, so the ROI is pretty good.

Also note the code to pick and insert the renderer. It’s not complex or anything, but it yields very convenient results. Here’s the output of running this one on the same source we used for the record:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79

/*global Ext */
/*global ACDRI */
/*global MTSI */
Ext.ns('ACDRI.ui.grid');

/**
* @class ACDRI.ui.grid.Customers
* @extends MTSI.ui.Grid
*/
ACDRI.ui.grid.Customers = Ext.extend(MTSI.ui.Grid, {
title: 'Customers',
record: ACDRI.record.Customer,
updateConfig: {
xtype: 'ACDRI.ui.form.Customer'
},
addConfig: {
xtype: 'ACDRI.ui.form.Customer',
},
initComponent: function () {
//this.sortInfo = {
// field: 'part_id',
// direction: 'asc'
//};
var config = {
url: 'customers',
itemName: 'Customer',
columns: [{
header: 'Comments',
dataIndex: 'comments',
sortable: true,
hidden: false,
width: 226
}, {
header: 'Color Code',
dataIndex: 'color_code',
sortable: true,
hidden: false,
width: 70
}, {
header: 'Id',
dataIndex: 'id',
sortable: true,
hidden: false,
width: 35
}, {
header: 'Last Edited Date',
dataIndex: 'last_edited_date',
sortable: true,
hidden: false,
renderer: ACDRI.fn.Renderers.dateTime,
width: 112
}, {
header: 'Name',
dataIndex: 'name',
sortable: true,
hidden: false,
width: 136
}, {
header: 'Price Approval Required',
dataIndex: 'price_approval_required',
sortable: true,
hidden: false,
renderer: ACDRI.fn.Renderers.bool,
width: 161
}, {
header: 'Sales Representative Id',
dataIndex: 'sales_representative_id',
sortable: true,
hidden: false,
width: 161
}]
};

Ext.apply(this, Ext.apply(this.initialConfig, config));
ACDRI.ui.grid.Customers.superclass.initComponent.apply(this, arguments);
}
});

Ext.reg('ACDRI.ui.grid.Customers', ACDRI.ui.grid.Customers);

Note that there are probably some things that we could do a little better. For instance, we should probably check for text fields and not include them in the grid. We could also be clever and look for anything ending with _id and have it hidden by default. But in general, the above is very nice results. At the very least you are probably going to need to reorder the columns (although you could give some kind of grid_order_id to the column definition in the Schema::Result code, but I don’t think that’s worth it.)

So I hope someone else enjoyed reading this as much as I did writing it. After I’ve exhausted all of the little Ext things I can use this for (form, etc) I will probably look into messing with DBICx::AutoDoc to add in Dia outputs.

2 Comments

Filed under: Uncategorized

Template.Tiny

Author: fREW Schmidt

25 Jan

(Sorry if you heard this already )

At $work we do as much “view” type code as we can in JavaScript with the ExtJS framework. I have personally found it to be a great framework to work with, although often it is lacking in the non-UI department. One thing that at first I really liked about Ext was their Template and XTemplate classes. But as time went on I got more and more annoyed with those modules.

I’ve always thought that Template-Toolkit was a really nice templating library to work with. I hate templating html because of all the weird little gotchas having to do with CSS and whatnot, but doing almost all that kind of work in JavaScript I have started missing Template-Toolkit. At some point I heard about Template::Tiny and this past Saturday I thought, “that’s like, 160 LOC…I could probably port that to javascript!”

So I did! It’s not really done yet, but the current code is at github. I need to finish porting the test suite from Perl 5 to JavaScript so that I can ensure correctness (I am certain I screwed up some stuff.)

Alias mentioned in his post that I use XRegExp (600 LOC) to help out with the Regular Expression support in JavaScript for this module. I actually wasn’t going to, since the only thing I needed it for was the /x flag, or to be more clear telling the parser to ignore whitespace, but I want to keep Template.Tiny in sync with the Perl version, and I really don’t want to strip out all the whitespace by hand. If someone takes issue with the dep they can fork away

So once it is fully ported I fully intend to use it entirely from now on instead of the Ext templating. But take note, Ext templates and Template.Tinies (what?) really solve different problems and have different goals. The following is a list of Pros and Cons of each:

Ext.(X)Template

Pros:

Can be compiled to JavaScript, for speed
Allows complex expressions in “if” blocks (see Example 1)
Interesting “topicalizing” feature (Example 2)
Crashes when you leave out a variable (no mystery as to why a field is blank) (Example 3)
Can execute arbitrary JavaScript code in a template for complex stuff (Example 4)
Neat automatic “current item” style variables when you are iterating over an array (Example 5)
Basic Math Support
Ability to call functions associated with Template object
Very cool builtin Renderer support (Example 6)

Cons:

No else if. If you want something like that you must do if !expr. Lame.
Crashes when you leave out a variable (mostly that’s just annoying)
This is almost entirely subjective, but having xml as a templating thing is kinda gross.
Not really open source, so I can’t use it in personal projects and have people use my code in a corporate setting

Example 1:

1
2
3
4
5
6
7
8
9
10
11
12

var data = { age: 21 };

// XTPL
var tpl = new Ext.XTemplate( "<tpl if='age > 18'>Can Vote!</tpl>");
var str = tpl.apply(data);

// TT
var tmp = new Template.Tiny();
var str = tmp.process(
"[% IF old_enough %]Can Vote![% END %]",
{ old_enough: (age > 18 )}
);

Example 2:

1
2
3
4
5
6
7
8
9
10
11

Example 3:

1
2
3
4
5
6
7
8
9
10
11
12

var data = { werld: 'fail'};

// XTPL
var tpl = new Ext.XTemplate( "Hello {world}");
var str = tpl.apply(data); // error message

// TT
var tmp = new Template.Tiny();
var str = tmp.process(
"Hello [% world %]",
data
); // silent failure

Example 4:

1
2
3
4
5
6
7
8
9
10
11
12

Example 5:

1
2
3
4
5
6
7
8
9
10
11
12
13

Example 6:

1
2
3
4
5
6
7
8
9
10
11
12

Template.Tiny

Pros:

Has IF/ELSE (Example 7)
Doesn’t crash on undefined fields
Nicely Licenced (Perl License)

Cons:

No complex expressions, math, or external javascript support
No topicalizing
Doesn’t crash on undefined fields (could be nice for debugging)
Probably slower (haven’t checked that yet)

Example 7:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

var data = { jack_slocum: 1 };

// TT
var tmp = new Template.Tiny();
var str = tmp.process(
"[% IF jack_slocum %]We don't appreciate else-ifs[% ELSE %] Woot![% END %]",
data
);

// XTPL
var tpl = new Ext.XTemplate(
'<tpl if="jack_slocum">We don\'t appreciate else-ifs</tpl><tpl if="!jack_slocum">Woot!</tpl>'
);
var str = tpl.apply(data);

I am planning on making a wrapper for TT for our stuff that will allow an anonymous function that will do data transformation like above. But as you can see above XTemplate really has more to offer, it just annoys the heck out of me on a regular basis

1 Comment

Filed under: Uncategorized

Writing vs. Writing

Author: fREW Schmidt

28 Oct

I enjoy updating this blog. Part of it is that I like writing, and part of it is that I kinda feel famous with all these great coders reading the words that I write. But I like programming better. That is why lately I’ve been posting less and coding more.

In posting modules to CPAN I’ve learned a lot of different things. First off, testing is easy. But before you test you have to set up your test environment. It’s not really hard, mostly it’s just a hassle. The obvious good thing about testing is that when I add features (my coworker asked for one for CGI::Application::Plugin::DBIx::Class just today) I know with pretty good certainty that I didn’t mess anything up, even if it’s a trivial change.

One problem I do have is that I consistently forget to add dependencies. The only think I can think to do is to have a vanilla perl install that I always test the release on. The problem with that though is that it is time consuming. If anyone has tips on how to deal with that let me know.

Another thing that I’ve learned is that Dist::Zilla is an invaluable tool for building releases. I love that it sets my version number, fills in my POD, sets up all my metadata, and even uploads my module to CPAN for me. It lets me spend less time learning about my build tool and more coding.

2 Comments

Filed under: Uncategorized

Brief Addendum: Send Email

Author: fREW Schmidt

26 Aug

Ok, so for some reason I left out sending email from last nights’ post. Here it is: sometimes people forget about RT, or they have so many RT’s that they don’t know which ones are fixed and broken. Well, a small nudge via email can convince them to fix a longstanding bug. Of course, if it’s reasonable sending a patch wouldn’t hurt either…

0 Comments

Filed under: Uncategorized

How to Help without being a Rockstar

Author: fREW Schmidt

25 Aug

I think a lot of people who use perl have the idea that to help the perl ecosystem be they must be rockstars who churn out exorbitant amounts of code that is well tested and well factored. That is just not true!

The easiest thing one can do to help out in the perl ecosystem is to create tickets for any issues you have with modules on RT. It’s not really that much of a hassle and it can help authors out a lot. Make sure that you include enough information for the authors to at least understand what went wrong. I won’t explain all that as my audience is comprised of programmers.

in that same vein, sometimes people forget about RT, or they have so many RT’s that they don’t know which ones are fixed and broken. Well, a small nudge via email can convince them to fix a longstanding bug. Of course, if it’s reasonable sending a patch wouldn’t hurt either…

Another thing which I’ve mentioned before is to use CPAN ratings. Again, this is super easy, but if the module works for you and you have no complaints, 5 stars can really tell other users a lot!

And then of course there is communication on IRC. Often on IRC, after a user asks for clarification, developers will ask for a doc patch. Although this is easy for the user to add (depending on the subject of course) it will help future users significantly.

How many RT’s have you submitted this week? CPANRatings? Doc Patches?

3 Comments

Filed under: Uncategorized

Perl 6 in Perl 5 FOR THE WIN

Author: fREW Schmidt

17 Aug

Today I wanted to generate a list from another list. Typically I would use map for this, but I wanted to iterate over two elements at a time, instead of one at a time. (A lot of people said to use natatime from List::MoreUtils, over and over. They didn’t read my question very carefully, especially since I specifically said I wanted natatime but with map.)

Anyway, mst pointed out Perl6::Gather, which works perfectly for this situation! Ah the beauty of Perl 6 in Perl 5. Here are the codez:

1
2
3
4
5
6
7
8
9

my @list = (
foo => [qw{bar baz biff}],
);

my @new_list = gather {
while (my ($foo, $bar) = splice(@list, 2, 0) ) {
take map { "$foo/$_" } @list;
}
};

So basically what that does is create an implicit accumulator and every time you call take it adds the arguments to take to the accumulator. In Perl 6 map can iterate over multiple items at once, so this would be silly in that context, but in Perl 5 it’s quite helpful!

0 Comments

Filed under: Uncategorized

CPAN Ratings Day

Author: fREW Schmidt

13 Aug

You may have noticed that there really aren’t a lot of CPAN Ratings out there currently, but you have a chance to help that. The past couple of weeks I’ve done two or three CPAN ratings every Thursday. Just go to CPAN Ratings, get an account, and rate modules that you are a fan of.

Generally criticizing modules in active development is a bad idea since bugs should really go to rt. But if the module is “done” and there are bugs, a bad review might be feasible.

Anyway, Happy CPAN Ratings day!!

2 Comments

Filed under: Uncategorized