I just released a new version of DBIx::Class::Helpers and it has two new components: DBIx::Class::Helper::ResultSet::ResultClassDWIM and DBIx::Class::Helper::Schema::GenerateSource.

Helper::ResultSet::ResultClassDWIM

This component solves an issue I’ve seen both by myself and with my coworkers; it’s too hard to remember/type the following:

So I wrote this component which will let you generically write:

or use the specially hardcoded:

Handy right?

Helper::Schema::GenerateSource

This component is a little more unusual. The idea is to take care of some of the issues I mentioned here. It doesn’t solve everything due to some design issues in DBIx::Class, which I hope to take care of soon. Basically the idea is that instead of the boilerplate files that you get when you use DBIx::Class::Helper::Row::SubClass, you can instead just do the following in your schema:

The main issue is that even though this correctly associates a new source to the schema, you cannot currently add relationships to the source. I’ll make another post when I fix that, but I doubt I’ll get it into the next release of DBIx::Class.

In other news I’m doing some pretty sweet stuff with the SQL generation code in DBIx::Class and I hope it will be ready for the next release. I’ll post more when that’s released.

Do you remember when you first realized that you were not the only person with a perspective in the world? I do. I was 5ish and I remember looking into the car to the left of me and seeing another person looking at me from their respective car. I remember thinking, “This is not what it is like from their point of view.” I distinctly remember reevaluating things all day that day. I am sure that I was still just as selfish and childish as I was before that moment, but it certainly changed my point of view.

Major features this has over DBIx::Class::Schema::Versioned:

• Multiple files for migrations
• Perl files in migrations
• Shared Perl/SQL for different databases
• Downgrades
• Not to mention extreme customizability

So try it out today! I am looking forward to getting bug reports soon

Oh also, if it does not quite do what you want…

PATCHES WELCOME!

I write a lot of ExtJS grids at work. I have written JavaScript classes for our Ext grids that generate as much as possible automatically, but the actual column definitions of the grids are almost always unique. The project I am on now is nearing our first real deploy, and we’re late, so things have been really, really busy.

It wasn’t until recently that I realized just how much time I spent working on grids and their related records (representation of the rows of a grid.) Although even if I’d known just how much I do this at the beginning of the project, I certainly didn’t know DBIx::Class as much as I do now, in addition to the other 4 non-core modules that I’ll mention.

Because I’ve been working all these super long hours (12-14 a day), often after a frustrating day I’ll try to code something more relaxing and rewarding at home or at work but not during work. So today I decided to finally take the plunge and do what I’ve been pondering for a few months and write some code to generate Ext scaffolding for me. So I’m going to walk you through the script that I wrote (and will probably work on more as time goes by.)

This isn’t a full module, not even in my repo yet, so it’s all just in one file. Here’s the boilerplate intro:

That’s pretty basic. We use a bunch of modules that I’ve used in at least 1 other project before and was happy with.

That’s the code to parse any kind of Catalyst config file and the grab a new schema based on it. It’s a ton of biolerplate but I live with it.

So next up is a basic inflection function that gives us all the different forms of a word we might need. It starts with either “single_foo” or “SingleBar” and gives us six variations based on that:

So that uses String::CamelCase and Lingua::EN::Inflect combined with join and split mostly.

We’re *almost* ready to generate a record. But first we need to define a mapping from the data type in the database to the data type that Ext uses:

Ok, now let’s look at the code to generate an Ext.record:

So what’s going on here is that we get the source from the schema. The source could be considered something like a table definition, although it can also point at a view or whatever too. Then we start generating the string representing our record, and then we use a join/gather combo to get the column data the way we want it.

We could certainly just use a more complex map instead of the for+gather that we have, but I personally feel than any map where you *must* use the block form is cumbersome. So we join together all of the strings that gather took, and then append the end of the definition, and voilà, we have a record!

Here’s example output on our Customer source:

If you’re still following along you should be fine with the next, much more impressive functionality.

I’d like to have the script do as much as possible to speed up my work, so let’s see how far we can take this. First, I made a mapping from type to renderer, so that things that are datetimes use our custom datetime renderer, same with booleans:

Next, I wrote a function that would try it’s best to guess how wide a column should be based on it’s header and the average width of the strings inside of it:

Now finally we have the grid function:

We see nearly the same structure here that we did with the record, except much more intricate. Notice the code to generate the width uses the magic constant 7. As I played with this I found that 7 seemed to work for the font that Ext uses by default (Arial?) Optimally I would actually use some kind of metrics package to ask it for the width of all of the strings that I generated in the function above and average that, instead of averaging character lengths. But this seems to work really, really well, so the ROI is pretty good.

Also note the code to pick and insert the renderer. It’s not complex or anything, but it yields very convenient results. Here’s the output of running this one on the same source we used for the record:

Note that there are probably some things that we could do a little better. For instance, we should probably check for text fields and not include them in the grid. We could also be clever and look for anything ending with _id and have it hidden by default. But in general, the above is very nice results. At the very least you are probably going to need to reorder the columns (although you could give some kind of grid_order_id to the column definition in the Schema::Result code, but I don’t think that’s worth it.)

So I hope someone else enjoyed reading this as much as I did writing it. After I’ve exhausted all of the little Ext things I can use this for (form, etc) I will probably look into messing with DBICx::AutoDoc to add in Dia outputs.

If I were to pick one of the modules that I’ve written so far to be my legacy DBIx::Class::Helpers would be it. Maybe later it will be DBIx::Exceptions, but as of now that’s technically vaporware.

So over the Christmas break I’ve been working on updating it a bit. First and foremost I added the exciting DBIx::Class::Helper::ResultSet::Union helper. The best use case (I can think of) for a union is when you want to get data from multiple tables as if they were one table. Here is how one might do something like that with the new module (example ripped from wikipedia):

I’d argue that if you have tables like the above you are Doin it Rong, but there are other times when it might make more sense, like if you wanted to maybe have some kind of autocompleter that works for more than one table (Artist, Album, Track) where all of the things have names and ids.

As you should be able to see from the name of the Component I have gone from naming things DBIx::Class::Helper::* to DBIx::Class::Helper::$namespace::*. Mostly that was because my esteemed DBIx::Class janitor, ribasushi, suggested it. I almost had done it before, but his asking for it was enough to get me to do it. The old names will be available for all of the 2.* series and the 3.* series with a warning. After that they will be removed.

Speaking of version numbers, I have also switched from using RJBS versions to using a more normal versioning scheme. Again, I wouldn’t take the credit for this one as it is mostly due to mst and ribasushi’s prodding. Basically the deal is that although RJBS versions make it dead easy for me to release code, they make it hard for the user to see what’s going on. How much has changed between releases? All a user knows is if a release breaks backcompat or not. So I’ll be using the more normal x.yyyzzw where x breaks backcompat, y means major new features, z means bugfixes, and w means minor fixes like pod or dist fixes. I imagine that even that could be automated by looking at current and previous release… Maybe next time

Next up is a much more complete DBIx::Class::Helper::ResultSet::Random which allows the selection of an arbitrary amount of rows, instead of just one. As mentioned in the POD I’ve only tested it on a large table with SQL Server (2005.) To be more explicit, if you are using mysql and it is as slow as people always say, show me a benchmark and we can work together to make it fast. The only thing I can think of to make it fast is to use RowNumberOver and do an IN for a list of shuffled numbers generated with perl. We’ll see though. It works fine for me right now so I don’t care

While I was updating all these little bits I went ahead and pregenerated the DDL for the schema, so I no longer depend on SQLT, which is actually surprisingly heavy. So that’s pretty sweet.

Lastly, String::CamelCase had a new release that no longer fails tests! So I’ve added it as a dep and removed the conditional load code. This isn’t as big a deal as most of the other stuff, but it is certainly a nice change.

Because of all of the major changes I’ve released the new version to cpan as a developer version, so please check it out. At the very least if someone finds a missing dep or something that would be good. If I somehow messed up the doc or something that would be good to find out as well. I plan on releasing for realz before (or on?) New years.

Blah blah blah perl marketing navel gazing wasting time blah blah blah perl is alive blah blah blah.

Ok, now that we’re done wasting time, here’s how to do something that (hopefully) will be useful!

I am working on a small Web Application in my increasingly rare spare time, and I decided I’d like to use OpenID for the authentication. Because of the structure of Catalyst applications this isn’t exactly easy as pie, but if you read this post it will be for you!

First off you have to install Catalyst::Authentication::Credential::OpenID (and dependencies) but wait! there are some issues you have to deal with first!

Catalyst::Authentication::Credential::OpenID depends on LWPx::ParanoidAgent, which has a few issues. See this RT for a patch that will solve it. When you run cpan (perl -MCPAN -eshell) run the command:

Wherever the directory is, save the yaml from the RT page to that directory.

Then install Catalyst::Authentication::Credential::OpenID the usual way and everything should work nicely. After the install is done you’ll need to configure Catalyst to use the module. Here is my config:

Note that I also have a realm for DBIx::Class. This is because I need to store actual data about the user and not just the fact that they have logged in with OpenID. I currently have my database set up such that a user can have more than one OpenID. I don’t have a UI for this, but I hope to add one eventually. Here are my DBIC Models:

That should be fairly obvious what it does, especially if you know DBIC already. Normally I leave my DBIC models pretty bare, but in this project I am generating the DB from the model, so I should make it as complete as possible.

Not really a lot going on there either.

This method, complete_openid_login, is really the flesh and blood of this login system. We call it after a person logs in with OpenID (see next section). The main thing to point out is that authenticate method call. It will set the $c->user object up to hold the correct User object, which is a good thing. A secondary thing to point out is that I am using perl’s vomitous exception handling for flow control. It’s just as valid to use regular if-else stuff, but I like the way this works conceptually. Hopefully as time goes on the string error messages can be replaced with actual Error objects. More on that in the coming months.

Most of the above is fairly unremarkable. We’ve got the login method which hopefully needs no explanation, and we’ve got the create method, which is what creates the DBIC model of the user. The vast majority of it is error handling. A lot of that could be taken care of with something like FormFu, but I haven’t crossed that bridge yet, so I can’t bring you across it either.

The one thing I should probably mention from the above is the captcha_check bit. I use ReCaptcha to ensure that my users are humans (for now I will discriminate against the machines!) It’s pretty easy to set up and it works quite nicely. I may post on that at some point too.

And lastly, here is what I came up with for the OpenID login screen. I used this Simple OpenID Selector to help with the logging in for the user. I’ve modeled mine off of what was used on StackOverflow.

That’s pretty basic. Nothing very special except for the research of where to get the js to do what I did.

And that’s it! You don’t have to email users to activate their account, you don’t have to worry about storing passwords securely, and they don’t have to remember another pair of credentials to use your site!

So this is probably old hat to those people who are already big on architecture or know a lot about design patterns, but I thought it was a pretty clever implementation of data security. Anyway, first I’ll start off with how I actually did it, and then maybe talk about it in the abstract.

So here’s the idea, I have a user, and that user should only be able to view a certain set of messages. The messages are linked to groups which the users are linked to. So users have groups, and then groups have messages. So to display the messages we do something like this:

And then you can use that kind of code to limit other things which would more easily cause security issues:

The fact that DBIC allows you to chain your searches is really what allows this kind of thing to happen. Of course, it could be emulated with most data structure based ORM’s by modifying the data structure that gets passed to the search or find method.

(I am pretty sure that you could do this just as easily with DBIx::Class::Schema::RestrictWithObject, but chaining off of user makes a lot of sense to me, so for now that’s how I’ll pull that off.)

Now before we get into a more general discussion I’d like to point out that because of DBIC’s implementation (and possible emulation of it already previously mentioned) this shouldn’t really be too much of a performance hit. Of course, the more related_resultset based chaining you do the more tables you are joining into the query, and that’s where you will start seeing performance issues.

Ok, so the general approach:

It seems to me that it wouldn’t be too hard to make a Highlander (Singleton) that would basically have methods for all of your ResultSet’s (or tables in SQL-talk.) It would contain any user credentials that are needed to get at any data. The idea would be to have it throw an exception if you were to try to instantiate it without all of the data needed to do your security stuff. Really that’s just good OO; any instantiated object should be complete.

Now I have to point out that this really isn’t a complete solution. My friend Fjord works on Birdstack and they need to support the hiding of specific columns, of specific rows, depending on a number of criteria. It’s possible that he could do this for birdstack, but that would end up making each optional column a join table, which would be slow and cumbersome. I don’t remember how he solved the issue, but I imagine that the best way to pull it off would be with a Highlander class that filtered each Result (row) coming from each ResultSet. I guess it would need to return specialize read-only classes or something.

One way or another, I think that no matter what, this fine grained control of public vs private data is going to be hard to manage and slow in a regular RDBMS. An object database might be able to handle it better, but I haven’t really thought much in that vein yet.

So with that I say to you peace be within your walls and security within your towers and racks!

Some of you probably know that I have some opinions, thoughts, and ideas. I actually started this blog because I wanted to write my own (can you guess what?) Manifesto. I chose to write it as a blog because I tend to change my mind. Ask some of my friends and family. They have all observed that I was going to be a math teacher, a psychologist, a biological engineer, a doctor, and a writer. (Take note: I am none of those things.)

I started programming in earnest about 10 years ago, when I purchased Programming Perl. I had done some basic, but I knew that real programmers used perl. I knew I would never use any other language. 6 years later I turned my back on perl when a professor introduced me to ruby. While in ruby-land I learned functional programming, what MVC was, what an ORM is, and the beauty of syntax (I still dig 5.times {…}). I knew that Rails was the One True way to program websites and that Prototype and Scriptaculous were the only way to program javascript.

Then 4 years later someone offered to pay me to write perl. I came back somewhat grudgingly, and I came extremely close to trying to write a certain project with rails. Fortunately (for my current dogmatism) my boss convinced me to stick with perl. Somewhere along the line I learned that perl 6 is truly being developed. I helped some and had some fun. I read the book currently titled The Passionate Programmer. After reading it I decided to start seriously look into switching from IIS to Apache.

After setting up Apache on my personal computer so I could have a useful error log I started researching ORM’s. I found that The One True ORM of any given language is DBIx::Class. I will never use another ORM as long as I live. I have posted about it a few times now. I’ll leave that at that.

Larry Wall says that the three programming virtues are laziness, impatience, and hubris. I agree with his conclusions. The first two often lead to code reuse. Code reuse is an excellent goal. Code reuse is what keeps my current codebase nimble and exciting to work on.

Part of code reuse means using libraries to help you get your job done. Did I mention DBIx::Class? Yeah, it helps me get my job done. Now, when I first started getting paid to code I was told that in a professional context, we don’t waste our time reinventing the wheel. Agreed! Let us not reinvent the wheel.

So instead of reinventing the wheel, we’ll purchase a library that does the job for us find some Open Source library that does (or almost does) the job for us. Before I got further I’d like to make a few points about Open Source software. (Also, let me remind you that I am Holden Caulfield right now so I may be lying, on purpose or accident.)

I do not use Open Source software because I desire or need freedom. My political friends tell me that I am spoiled for saying that freedom is not the highest virtue and that I would not be willing to die for freedom. There are other virtues that I (hope) would be willing to die for, but that’s another chapter.

I do not use Open Source software because I am poor. I purchase indie video games because they are awesome works of art and they are not cheap. I donate to Open Source and otherwise free software that I regularly use because I am glad to pay for the excellent work that someone will do to make my job/life easier. I think that it is fine to ransom features as an Open Source programmer.

I do no use Open Source because I am a communist. There is no reason that a programmer should give you his time and effort for free. Let me redact that statement: there is no reason that a person should give you his time and effort for free. If you view me as a carbon offset to the earth and that everything I do should be given to the poor, that’s fine. We are all wrong sometimes. Let me be clear: I love Ayn Rand as a phiosophess and I agree with her unconditionally.

I use Open Source software because I am a programmer. Jeff Atwood says that “If it’s a core business function, write that code yourself, no matter what.“. I agree Jeff! The problem comes when you purchase an over-the-counter library, it suits your fancy perfectly, and then six months later, as always happens, the customer wants more. The library no longer works for you, so you either pay the Closed Source vendor to implement the features you need, or find another library and port all of your code to that.

This is what happens to me: I use an Open Source library that does what I need. I eventually outgrow it or it doesn’t meet a specific need, I either whine enough to get someone to add the feature I need, or I figure out how to add it myself. I’m not even a very good programmer; I just really like to program.

Let me put it another way: do you have any friends who really like to work on their car? Do they buy the brand new drive-by-wire automatic Toyota that is more black box than car?

This post is pushing up against the thousand word mark and we certainly wouldn’t want to go there, so I’ll repeat myself one more time: I am a programmer. I will continue to use Open Source software because I love to program and because I don’t want any Golden Handcuffs.

So programmers, ask and it will be given to you; seek and you will find; untar and the code will be opened to you. Suits: feel free to purchase black boxes as Golden Handcuffs. Thank you and have a nice weekend.

Recently (6 monthsish ago) I decided on an ORM to use at $work. It was pretty hard to make a decision because I’d never really used an ORM for a significant amount of time. Now that I am pretty confident with my chosen ORM I feel like I can make a more informed comparison.

I’m going to skip over the basics of declaring classes themselves. Often when researching ORM’s this is the main thing that people look at. Unfortunately it’s (in my opinion) not that important. As long as everything you want to do is supported, the base model class should just stay out of your way. Recently there have been complaints about the aesthetic appeal of things like DBIC. I prefer to look at conceptual beauty rather than syntactic.

I’d rather focus on major differences in underlying structure, and more importantly, how searches work. I do a lot more searches than I do anything else, and I’d bet that’s the same for you, after you do more than just the basic structure of your app.

So without further ado, the contenders.

Class::DBI

26 releases in 5 years. The most recent is from 2007.
2 authors, 33 credited.
The oldest of the discussed ORMs.

Searches are extremely simple, being limited to == and LIKE queries. Here’s an example:

You can do more complex things, but it’s really just writing SQL and giving that SQL search a name. A SQL dictionary approach you might say. Also note that the above returns an entire array of results, which is Not Great. You can use an iterator though for performance….but it still pulls it all into memory; it just doesn’t instantiate the objects right away, which is a little better, but still Bad.

A cool feature CDBI has is triggers for the lifecycle of the object. The triggers listed in the docs are:

1. before_create (also used for deflation)
2. after_create
3. before_set_$column (also used by add_constraint)
4. after_set_$column (also used for inflation and by has_a)
5. before_update (also used for deflation and by might_have)
6. after_update
7. before_delete
8. after_delete
9. select (also used for inflation and by construct and _flesh)

CDBI also has built in constraints, so you can do validation in your model.

Both of these can be done with regular OO in all of the other ORM’s, but having a predefined naming scheme for things like this helps people to quickly learn what’s going on.

Rose::DB::Object

70 releases in 3 years. The most recent is two months ago.
1 author, 11 credited.

Written with speed in mind. Be aware that because of these manual optimizations the code is harder to maintain. I was told this by one of the contributors to the project. But you do get very good speed (supposedly, I haven’t done any tests myself) because of it.

Here’s a basic Rose::DB::Object search. It returns an arrayref, which isn’t optimal, but you can get an iterator just as easily.

From my perusing of the docs it seems that Rose basically has all of the perl data-structure based searches that one would hope for in an ORM that abstracts away most SQL.

DBIx::Class

62 releases in almost four years. Most recent being days old.
One “author,” 69 credited.
It is very much made for the convenience of the programmer. The .09 series will be Moose-based. See slides for proof.

Here’s an example of a relatively complex search with DBIx::Class.

What I think is really great about DBIx::Class is the fact that you can chain searches. I really dig this feature. It lets me do things like this:

And most importantly, this is a single SQL query, not four.

I also need to mention that the DBIx::Class people have really helped me help them, which is not just a good feeling, I use the features I’ve added in production code. I can say for certain that working on their codebase has made me a better OO programmer.

Fey::ORM

22 releases in just over a year.
One author.
This is the newest of the four ORM’s reviewed here. Cut it some slack if it doesn’t have all the features that the other ones have.

Fey::ORM is made for people who are alright with actually writing SQL. It’s very … OO-y. For example:

What I think is really cool about Fey::ORM is that it has a standard method for creating relationships other than the usual has_one/has_many/many_to_many. Who wouldn’t want to be able to make relationships based on something other than equality? (Seriously though, I’d be able to use that at $work.)

So that’s what I gathered from a couple of hours of reading docs on CPAN and a few months of DBIC usage. I really like DBIx::Class, but I can see why people would choose some of the other ORMs. It does seem to me that the only reason to use Class::DBI is that you already have a giant codebase written on top of it. But if that’s the case, you could just use DBIx::Class’s compatibility layer…

Anyway, hope this was helpful for someone!

I’ve been told numerous times by people that I believe are smarter than me that I should do validation in my controllers and not my models. mst said that some validation, like low level primary key type stuff, can be in models, because it has to be. But if I recall correctly almost everyone was against validating things like email addresses in my models.

I just read this article and a lot of what alias says seems to make good sense to me. But if what he says is true that means it would be best if I validated as much as possible in my model, and then bubbled up any errors to the controller via exceptions or something like that. That would also make it simpler for me to generalize things that need to be a certain type value on creation, but should never change after creation.

So tell me, dear internet, why shouldn’t I validate as much as possible in my models?

I’ve used Open Source for a little over ten years now. I’ve been sufficiently indoctrinated that Open Source (Free Software) is both morally and technically the right choice. That’s not what this post is about. If you disagree with those premises, that’s fine. The idea here is that I use all kinds of Free Software all the time. I use Vim for a text editor. I use zsh as a shell. Firefox is my browser. This blog runs on WordPress. The webserver we use at work is Apache. And the of course all of our code depends on Perl and numerous libraries.

We don’t pay for any of that software! Not a dime! And that’s fine, but nothing comes for free.

So far I’ve worked on three open source projects. The first was TOME, a book sharing system we used at school. Next I wrote some of the spec tests for Perl 6. And then most recently I’ve been doing some things for DBIx::Class.

One of the excellent things about the DBIx::Class developer community is that they really do their best to help you to work on the source.yourself. Recently they (or more specifically ribasushi) helped me add the full sorting capabilities to the SQL Server parts of DBIx::Class. More lately I’ve been adding things for the paging capabilities, which is great because paging with SQL Server is horrendous.

Anyway, the most important part of all of this is that I am part of something that will help me and other people. Furthermore, it really wasn’t that hard to add the code. They showed me where to add it, did a little code review to help me clean it up, and that was it! If only more communities were like that the Open Source world might be even more vibrant.