A Foolish Manifesto

fREWdiculous!

Archive for August, 2011

Pages (2): « 1 2

You should be using git grep

Author: fREW Schmidt

13 Aug

Usually when searching through files I use ack which is an awesome tool indeed.

Unfortunately, though ack does indeed work on windows, using it on windows is a painful experience. The main two problems are that it’s slow and the color coding doesn’t work.

I figured I’d try out git grep, with the hope that it might be marginally better. I try my best to at least be familiar with all the git commands, so this is one of those things I had been meaning to do anyway.

I was in for a pleasant surprise when I found that git-grep’s color coding worked and it is extremely fast. I profiled it on linux and I get about an order of magnitude difference for a simple repo. If you use a repo with submodules the difference is even more visible, though possibly you would actually want to search submodules by default.

There are two features of ack that I use regularly. The first is that it doesn’t search files it shouldn’t, like .svn and .git. With git grep that is also the case, though it only works with git checkouts (which is all I have at this point.) The second feature is the ability to do –perl or –js or –java or whatever else. This can also be done easily with git grep, and I may make a bash wrapper called git-ack that just gives the interface of ack with the backend of git grep.

1
2
  ack frew --js
  git grep frew -- '*.js'

Sure, it’s marginally longer, but if I do make the wrapper it will just be ‘git ack frew –js’ and given that I have cool shell settings based on the directory I’m in I will set it up such that ‘ack frew –js’ just runs ‘git ack frew –js’

Also, check out the amazing -p argument for git grep; it uses git’s code parsing to include the function that matches are in.

  • 2 Comments
  • Filed under: Uncategorized

    • Powerful benchmarking with Perl and ab

    Author: fREW Schmidt

    11 Aug

    One of my projects at work was to make an SMS (and voice actually) gateway. The gist is that instead of our customers each having an account with whatever text message company, they go through us. The benefit is that with a larger pool of users for the text messages users can have a lot more flexibility with how they use their messages. Most gateways sell you messages per month, and we sell yearly messages.

    One of the major uses of our software is for duress; that is, sending text messages to all the students at a college in an emergency (note: sending SMS in an emergency is a really bad idea, but people want to do it …) Because of this we really want to put a premium on how many we can send at a time. Our old gateway (non-persistent Perl, weird database, bad API) was excruciatingly slow.

    To test the speed of our server when sending a large number of messages from a single server I wrote the following script to test a number of different situations.

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    #!/usr/bin/env perl

    use strict;
    use warnings;

    use JSON;

    use Getopt::Long::Descriptive;

    my ($opt, $usage) = describe_options(
      'benchmark.pl %o',
      [ 'concurrent|c=i', 'number of concurrent connections', {default => 5 } ],
      [ 'destinations|d=i',   "number of destinations to submit", { default => 100 } ],
      [ 'total-iterations|n=i',   "number of iterations to run", { default => 100 } ],
      [],
      [ 'help|h|?',       'print usage message and exit' ],
    );

    print($usage->text), exit if $opt->help;

    open my $fh, '>', 'testtest';
    print {$fh} to_json({
       message    => 'HELP! BUILDING IS ON FIRE!',
       destinations => [map +{
          phone_number => 1000000000 + $_,
          child_id     => $_,
       }, 1..($opt->destinations)]
    });

    system(
       'ab',
       qw(-T application/json),
       '-n' => $opt->total_iterations,
       qw(-p testtest),
       '-c' => $opt->concurrent,
       'http://10.6.1.56:3000/api/1/test/sms'
    );

    So we have a handful of nice commandline options, we generate a file of JSON, and then we have ab run the actual speed test.

    One of the really neat things you can do is have perl run more than one ab instance at a time, this allowing you to test multiple urls, which ab doesn’t support natively.

    Anyway, good luck speed testing the hotspots in your app!

  • 1 Comment
  • Filed under: Uncategorized

    • New Stuff in Class::C3::Componentised 1.001000

    Author: fREW Schmidt

    9 Aug

    I’m very excited to finally announce a feature that I’ve toyed with in Class::C3::Componentised for over a year now.

    New in the current release of Class::C3::Compontised is Class::C3::Componentised::ApplyHooks. The gist is that you can run code, or more importantly methods, against the class being injected into.

    I wouldn’t be surprised if few people reading this actually know what Class::C3::Componentised is actually used in; the answer is DBIx::Class. The upshot of this new feature is that you could write a component to add columns or relationships much more nicely than before.

    A simple example and usage of it is as follows:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    package MyApp::Schema::ResultComponents::TimeStamp;

    use base 'DBIx::Class::TimeStamp';

    use Class::C3::Componentised::ApplyHooks;

    AFTER_APPLY {
      my $class = shift;

      $class->add_columns(
        timestamp => {
          data_type     => 'datetime',
          set_on_create => 1,
        }
      );
    }

    1;

    package MyApp::Schema::Result::User;

    use base 'DBIx::Class::Core';

    __PACKAGE__->table('users');

    __PACKAGE__->load_components('+MyApp::Schema::ResultComponents::TimeStamp');

    ...;

    1;

    There are two caveats.

    You cannot simply “use base” a component with Apply Hooks. The hooks just won’t run. There are ideas on how to solve that, but maybe that’s a feature.

    The second is that you must load_components after table is set. The simple solution at this point is to yuse DBIx::Class::Candy, as it runs the table code at compile time. I hear that another solution is in the mix as well, which will basically make add_columns et al create the underlying result source on demand, but we’ll see if that materializes.

    Either way, it will be interesting to be able to use this internally as it easily lets me share table definition data in a cleaner way than I did before. Hopefully you find it useful as well.

  • 1 Comment
  • Filed under: Uncategorized

    • Weekly Status Report 1

    Author: fREW Schmidt

    8 Aug

    This week I

  • 0 Comments
  • Filed under: Uncategorized

    • New Stuff in Log::Contextual 0.004000

    Author: fREW Schmidt

    6 Aug

    I just released Log::Contextual 0.004000 and it has a handful of great features.

    It now supports arbitrary levels, so where before you simply had:

    • trace
    • debug
    • info
    • warn
    • error
    • fatal

    Now you can have any levels by just saying

    1
    use Log::Contextual -levels => [qw(lol wut zomg)], ':log';

    which would import functions for log levels lol, wut, and zomg.

    But the really exciting thing is that now you can make a base class of Log::Contextual and set defaults for all of the different import options:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    package MyApp::Log::Contextual;

    use parent 'Log::Contextual';

    use Log::Log4perl ':easy';
    Log::Log4perl->easy_init($DEBUG);

    sub arg_logger { $_[1] || Log::Log4perl->get_logger }

    1;

    The $_[1] in arg_logger is whatever logger the user passed in when they said “use MyApp::Log::Contextual -logger => …”. You can choose to allow them to override the logger like I did above, or you can force them to always use the logger that you set.

    You can also use the arg_levels, arg_default_logger, and arg_package_logger methods, but I doubt anything other than arg_logger and arg_levels will be common.

    Anyway, I hope people find this release as exciting as I do; I know it will make my code a lot nicer.

  • 1 Comment
  • Filed under: Uncategorized

    • DBIx::Class Extended Relationships

    Author: fREW Schmidt

    4 Aug

    Since the dawn of time DBIx::Class relationships were simply a set of columns related to each other via equality. For the most part this is good enough, but DBIx::Class aims at 100% power for all databases (unlike some other ORMs… :-) .)

    In May what we internally called “extended relationships” was added to DBIx::Class. (docs here) Basically this allows you to use the full power of SQL::Abstract to define your join conditions. Just today I finally had a chance to use it. My join condition is simply “tableA.user = tableB.user OR tableA.shared = 1″. Here is how I defined it:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    has_many output_devices => '::OutputDevice', sub {
       my $args = shift;
       
       my $shared = { "$args->{foreign_alias}.shared" => 1 };
       return ([
          { "$args->{foreign_alias}.user" =>
             { -ident => "$args->{self_alias}.user" } },
          $shared,
       ],
       $args->{self_rowobj} && [
          { "$args->{foreign_alias}.user" => $args->{self_rowobj}->user },
          $shared,
       ]);
    };

    There are two interesting things to point out.

    It’s a coderef. This is so that we can parameterize various salient bits, like the aliases for the tables and, if it exists, the row object. There are a few other things passed, but I didn’t need them here.

    The other interesting thing is that instead of returning a single join condition, I returned two conditions. The first is simply a join condition, which might be used if you were to call search or related_resultset. The second is a where clause to allow more basic SQL to be used when you already know one side of the join clause. So instead of doing something silly like “JOIN Foo ON Foo.bar = me.bar WHERE me.bar = 1″ we can simply use “FROM Foo where Foo.bar = 1″. Also note that a handful of helpful methods in DBIx::Class (create_related and friends) require the second condition to be defined and fairly simple.

    It is definitely more verbose to use join conditions like this, but having the ability is great.

  • 2 Comments
  • Filed under: Uncategorized

    • Event Loops are better than while (1)

    Author: fREW Schmidt

    2 Aug

    One of the projects that I worked on last year had a number, five I think, of background daemons. Basically the way we implemented this was by making a DoesRun role that looked something like the following:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    package Lynx::SMS::DoesRun;

    use Moose::Role;

    requires 'single_run';

    has period => (
       is => 'ro',
       required => 1,
    );

    sub run {
       my $self = shift;
       while (1) {
          $self->single_run;
          sleep $self->period;
       }
    }

    no Moose::Role;

    1;

    And then a typical Runner class looked something like this:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    package Lynx::SMS::Runner::Voice;

    use Moose;
    use Log::Contextual::SimpleLogger;
    use Log::Contextual qw( :dlog :log ),
      -default_logger => Log::Contextual::SimpleLogger->new({ levels => [qw( warn error fatal )]});

    with 'Lynx::SMS::DoesRun';
    has schema => (
       is => 'ro',
       required => 1,
    );

    sub single_run {
       my $self = shift;

       log_debug { 'Processing voice messages' };
       my $guard = $self->schema->txn_scope_guard;
       while ($self->schema->resultset('MessageChild')->voice->unsent->not_blocked->count) {
          ...
       }
       $guard->commit;
    }


    no Moose;

    __PACKAGE__->meta->make_immutable;

    1;

    And lastly, a script using the runner looked like this:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    #!/usr/bin/env perl

    use 5.12.1;
    use warnings;
    use rlib;

    use Lynx::SMS::Runner::Voice;
    use Lynx::SMS::Schema;
    use Config::ZOMG;

    my $config = Config::ZOMG->open(
       name => 'Lynx::SMS',
       path => '.',
    );

    my $voicer = Lynx::SMS::Runner::Voice->new(
       schema => Lynx::SMS::Schema->connect( $config->{Model}{DB}{connect_info} ),
       period => 60, # seconds
    );

    $voicer->run;

    Anyway, that was all well and good, but at some point things would die and the whole thing would come crashing down, so then we started adding an eval around the call to run in the script, and then I thought, “someone must have done this before…” So I asked in the #catalyst channel on irc.perl.org and rafl pointed out that this is what event loops (POE being the oldest and probably most popular) are great at.

    So I updated the run method in the DoesRun role, so now it looks like this:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    sub run {
       my $self = shift;
       my $j = AnyEvent->condvar;
       my $w = AnyEvent->timer(
          interval => $self->period,
          cb => sub { $self->single_run },
       );
       $j->recv;
    }

    Ok, cool enough, it basically does the exact same thing as before except it never dies. But then I had an idea, on a server with 16 Gigs of RAM and a dual quad-core CPU five fat perl daemons is hardly an issue. But when developing it’s certainly a hassle to have to start them all up myself. So why not combine them and have them all run in the same process? Cake! I made the following Runner class to do the magic:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    package Lynx::SMS::Runner;

    use Moose;

    has tasks => (
       is => 'ro',
       default => sub { [] },
    );

    sub run {
       my $self = shift;
       my $j = AnyEvent->condvar;

       my $x = 0;
       my @tasks = @{$self->tasks};
       @tasks = map {
          my $task = $_;
          AnyEvent->timer(
             after    => ($x++ / @tasks),
             interval => $task->period,
             cb       => sub { $task->single_run },
          )
       } @tasks;
       $j->recv;
    }

    no Moose;

    __PACKAGE__->meta->make_immutable;

    1;

    The after thing is weird, but the idea there is that each task will start at a different time, so things are more likely to run at a different time. Not really important, but it makes the logs easier to follow for me.

    And then here is my script using it:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    #!/usr/bin/env perl

    use 5.12.1;
    use warnings;
    use rlib;

    use Lynx::SMS::Runner::SMS;
    use Lynx::SMS::Runner::Voice;
    use Lynx::SMS::Runner::Emailer;
    use Lynx::SMS::Runner::Notifier;
    use Lynx::SMS::Runner;
    use Lynx::SMS::Schema;
    use Config::ZOMG;
    use Lynx::SMS::Logger;
    use Log::Contextual
      -logger => Lynx::SMS::Logger->new({
        levels_upto => 'trace',
        format      => '[%d] %m',
      });

    my $config = Config::ZOMG->open(
       name => 'Lynx::SMS',
       path => '.',
    );

    my $schema = Lynx::SMS::Schema->connect($config->{Model}{DB}{connect_info});
    my $runner = Lynx::SMS::Runner->new(
       tasks => [
          Lynx::SMS::Runner::SMS->new(
             schema => $schema,
             period => 1, # seconds
          ),
          Lynx::SMS::Runner::Voice->new(
             schema => $schema,
             period => 1, # seconds
          ),
          Lynx::SMS::Runner::Emailer->new(
             schema => $schema,
             period => 60*5, # 5 minutes
          ),
          Lynx::SMS::Runner::Notifier->new(
             schema => $schema,
             period => 60*60*24, # 1 day
          ),
       ]
    );

    $runner->run;

    One thing that would improve this whole thing would be to capture dies or whatever and log $@ in our standard error logger thing. I haven’t quite figured out how to do that yet, but if someone knows how and comments I’d appreciate it.

  • 2 Comments
  • Filed under: Uncategorized

    • Getting More Done

    Author: fREW Schmidt

    1 Aug

    Today I purchased 59 Seconds, recommended by Jeff Atwood. I struggle with procrastination as much as anyone else so I’m willing to spend 10 bucks to try to get more done. The author recommends four things to attain a given goal:

    • plan well
    • reward yourself
    • focus on benefits
    • tell people

    I’ve kinda slacked off with Open Source stuff the past year (see the graph at metacpan) and I’d like to remedy that.

    Goal: Participate more

    My overall goal is to participate more (more blogging, more patches/pull requests, more releases and bug fixes for my own modules, etc.)

    The book recommends breaking the overall goal into five for fewer distinct subgoals that are measurable and whatnot, so here are my subgoals:

    Subgoal 1: Do two, public, permanent code related things a week

    For example submit a patch or release a distribution. Pushing to github isn’t permanent :-)

    I think this is pretty feasible. I have a huge backlog of things to do (which I’ll post, semi-prioritised at the end of this post) and I already set aside time on Mondays and Fridays to get stuff done.

    To get this done I’ll work on stuff 2-3 hours on Monday night and 2-3 hours Friday night (assuming nothing is going on.) I also might do stuff Tuesday or Wednesday, but time is scarce with wedding planning. I’ll work on the most important things first, and after that’s done I’ll work on the most fun stuff; the idea being that I can get two things done in a week that way.

    I’ll count this as achieved (though I don’t want to stop) after I’ve done two public things a week for two months straight. I’ll post weekly progress reports every Monday when I start hacking about the previous week.

    My reward for finishing this is to buy myself a new car stereo.

    Subgoal 2: Post two tech related posts to my blog twice a week

    Again, this sounds reasonable. I have a long list of half finished posts that I should just clean up and post. These should keep me busy for a while.

    I’ll have a similar plan for this as I do for Subgoal 1; instead of Monday and Friday I’ll do Tuesday and Sunday. I’m a little nervous about those days as the are mostly booked for the foreseeable future, but what I hope to do is use remaining time on Mondays and Fridays to work on blogging.

    I’ll count this as achieved after I’ve done two tech blog posts a week for eight weeks straight. Weekly progress shouldn’t be necesary :-)

    My reward for finishing this is to buy myself new car speakers.

    Subgoal 3: Get used to using some bug tracker for Open Source work

    Currently I am not very good about using a bug tracker for anything other than work. I’ve used Google’s bug tracker, github’s issue tracker, RT, and am currently toying with ticgit. I need to pick something, get good at it, and stick with it.

    I think I can do this. It mostly takes discipline to pull this off. I need to just add stuff to whatever bug tracker every time I am going to add a new feature, fix a bug, clean stuff up, whatever. I think I’m gonna use ticgit for now since I don’t want to write my own bug tracker yet.

    I’ll count this achieved after every single one of the things from Subgoal 1 that are my own modules use the bugtracker I’ve chosen.

    My reward for finishing this is to buy myself an amp for my car stereo.

    Benefits

    In the book he says you should state three benefits that you would get from finishing your goal. My three benefits are: I would be more respected in the community; my life would be easier as I’d have the things that I’ve been wanting for a while; and lastly, I could get more organized and more quickly ramp up on doing things.

    TODO list

    • DBIx-Class-Helpers – Jnap’s stuff
    • DBIx-Class – Date Math stuff
    • Log-Contextual – subclassable importer
    • DBIx-Class-Helpers – StateHook
    • DBIx-Exceptions – Plan out remaining bits to get done
    • DBIx-Class-Shadow – Plan out remaining bits to get done
    • Git-Conversion-Book – Table of Contents
    • Git-Conversion-Book – Each Chapter (x10?)
    • Object-Exporter – Plan out remaining bits to get done
    • Log-Location – Plan out remaining bits to get done
    • HTML-Zoom-Widgets – Plan out remaining bits to get done
    • Catalyst-ActionRole-PseudoCache – Optionally use a real cache
    • DBIx-Class-Schema-Auth – Plan out remaining bits to get done
    • Net-Goodreads – Plan out remaining bits to get done
    • Data-Dumper-Concise – DwarnOnly
    • DBIx-Class – Logger Configuration
    • DBIx-Class – Logger Console Width
    • DBIx-Class – Non-Result based FilterColumn
    • DBIx-Class – Instance Based Relationship Helpers
    • DBIx-Class – Join Args
    • DBIx-Class – ResultSet Components
    • DBIx-Class-DeploymentHander – code cleanup
    • DBIx-Class-DeploymentHander – various doc patches
    • DBIx-Class-Helpers – Fix remove columns
    • DBIx-Class-Helpers – Work way to make SQLMaker helper
    • DBIx-Class-Helpers – ReadOnly
    • Log-Contextual – NullLogger ?
    • git-super-status – More Filters
    • git-super-status – More Sorts
    • git-super-status – Configurable Columns
    • git-super-status – Configurable Colors
    • git-super-fetch – get it started?
    • SQL-Translator – Finish SQL-Server cleanup
    • teatime – Convert 100% to REST
    • try-awesome – We have plenty exception generating frameworks, now we need a catching one
    • Log-Contextual-P6 – Port LC to P6
    • Log-Contextual-JS – Port LC to JS

    Pending Blog Posts

    • Event Loops are better than while(1)
    • Powerful benchmarking with Perl and ab
    • Commercial Cross Platform Git
    • Alternatives to Sub::Exporter
    • Cool things you can do with Sub::Exporter
    • Why Exceptions are here to Stay
    • The Problem with Exceptions
    • Refactoring dispatch tables into objects
    • Emacs vs Vim
    • Simple Perl Proxy for Productivity
    • Sweet Code, thanks to Moose and Catalyst
    • Keep track of failing tests
    • Revised OpenID post (or not)
    • Write Code to be Cargo Culted
    • The Toolsmith: Vim, zsh, IRSSI, XMonad, etc etc
    • Filtering out things
    • On Syntactic Arcana
  • 1 Comment
  • Filed under: Uncategorized
    • Pages (2): « 1 2

    PhotoStream

      IMG_9370IMG_9363IMG_9360bwIMG_9360IMG_9359bwIMG_9359IMG_9357IMG_9355IMG_9353bw
    August 2011
    S M T W T F S
    « Jul   Sep »
     123456
    78910111213
    14151617181920
    21222324252627
    28293031  

    Links

      Archives

      Meta

      Copyright © 2007 - A Foolish Manifesto - is proudly powered by WordPress
      Nearly all content related to SuperCollider Programming and Perl Programming.
      Illacrimo Theme is created by: Design Disease brought to you by LifeSpy.