Perl Core Syntax Wishlist: Role Support

I want to see Role's added, even PHP got Traits before Perl. It doesn't have to be a huge thing, in fact all I want is the composition aspect. Let me do this: I don't really want or need anything else right now, just that would be fine. We should have interfaces too, but they aren't really required just to support Role's. We should probably have some sugar like I did for class (e.g. role keyword. and some of the same modules/pragmas loaded for this too)

New Module: MooseX::RemoteHelper (RFC)

Background

I have spent much of the last year writing and refining Remote Facades. At this point I've worked with SOAP, REST/JSON, and RPC url-form-encoded API's. One of the hardest parts I've found is dealing with the serialization of a Data Transfer Object with a Perl interface into whatever the remote is expecting. When I started I didn't know of these patterns, or really anything about these patterns. I highly recommend reading Patterns of Enterprise Application Architecture) if you want to know more about these patterns or things like Active Record and MVC.

problem

The problems I've encountered are many, including the fact most remotes are buggy or have a cludgy interface. Though there's nothing you can do about a remote api that you don't control, you can make your local API much cleaner and more native. Doing this however comes with a few challenges. One is that you have to map a local attribute name to a remote attribute name, because Perl uses underscores, and Java uses camel case, e.g postal_code and postalCode. The second problem is that many times the value of the attribute in its perl native form is not what the remote wants, e.g. perl boolean "1" remote "Y", or a DateTime object to W3C formatting. This second is not quite the same as mapping, because mapping is one to one, this translation could be turning an array into a comma separated string. The third problem, I didn't run into until after I "solved" the first , is how should I deal with nested complex objects (one's that can't be just converted to just a string).

My first naive remote facade was very procedural and simply assembled top to bottom, in part because it was based on SOAP::Lite, and in part because I had yet to figure out a better way. This resulted in a giant unwieldy if/then chain. Obviously my translations were just inline too.

The next thing I tried was using triggers to construct a request hash to pass to XML::Compile::SOAP. This worked better as the hash constructing code was kept right next to the attribute, so if I needed to modify the local or remote attribute, I could just go look at the attribute and the trigger tied to it.

After that I tried to use a map to translate from the native attribute name to the remote attribute name. This may have been more successful had it worked more like the Assembler in the Remote Facade. But ult imately since we were developing a rapidly changing API it seemed to bog me down, this is because I was changing the attributes on both sides of the mapping and thus the mapping at the same time (so at least 3 places). Here I was just manually dealing with the translation from a W3C DateTime format to the object I needed.

a solution

When I got assigned to yet another API and found myself doing yet another mapping and translation I decided that I needed to solve the problem. Enter the first iteration of MooseX::RemoteHelper. The first tie I used it with the form-url-encoded API so it was only needed for a single level of key, value pairs. .

Once I determined how to create MX::RemoteHelper it was simply a matter of using Class::MOP::Class API's to iterate all the attributes. The source of CompositeSerialization will give you some idea of how I did this.

Then I went back to apply this to a previous module, because the technique appears to be cleaner. Unfortunately I ran into a problem, this other API was a complex data structure, and how best to provide nested hashrefs and arrayrefs. Though I was now armed with Patterns I didn't know of one that would solve the problem. Fortunately a quick flip through the Gang of Four brought me to the Composite Pattern. I used this to write the recursive CompositeSerialization so that if I had a sufficiently complex nested structure I could simply create another object to deal with that. Here's a full example:

use 5.014;
use warnings;
use Data::Dumper;

package MessagePart {
    use Moose;
    use MooseX::RemoteHelper;
    with 'MooseX::RemoteHelper::CompositeSerialization';

    has array => (
        remote_name => 'SomeColonDelimitedArray',
        isa      => 'ArrayRef',
        is        => 'ro',
        serializer => sub {
            my ( $attr, $instance ) = @_;
            return join( ':', @{ $attr->get_value( $instance ) } );
        },
    );

    __PACKAGE__->meta->make_immutable;
}
    
package Message {
    use Moose;
    use MooseX::RemoteHelper;

    with 'MooseX::RemoteHelper::CompositeSerialization';

    has bool => (
        remote_name => 'Boolean',
        isa      => 'Bool',
        is        => 'ro',
        serializer => sub {
            my ( $attr, $instance ) = @_;
            return $attr->get_value( $instance ) ? 'Y' : 'N';
        },

    );

    has foo_bar => (
        remote_name => 'FooBar',
        isa      => 'Str',
        is        => 'ro',
    );

    has part => (
        isa      => 'MessagePart',
        remote_name => 'MyMessagePart',
        is        => 'ro',
    );

    __PACKAGE__->meta->make_immutable;
}

my $message
= Message->new({
    bool    => 0,
    foo_bar => 'Baz',
    part    => MessagePart->new({ array => [ qw( 1 2 3 4 ) ] }),
});

say Dumper $message->serialize

Which should give you this data structure:

Request For Comment

I've recently released a Trial version of MooseX::RemoteHelper to CPAN. I'm currently refactoring Business::CyberSource to use it, and it appears to be solid. What I'd like to know what people think of the module names, method names and any other comments they might have. I haven't been entirely sure that I've been naming things correctly while writing this, or that the code couldn't be better in other ways. If there's functionality you wish it had but doesn't let me know.

Perl Core Syntax Wishlist: Class support

I would like to see the class keyword become part of Perl, but unlike some I don't want it simply because it's nicer syntax. I'd like it to behave differently from package. I'd basically like to see this I'm sure smarter people than I could think of a few more things that might be nice to have by default in all classes. I would like to note that method signatures is not that big of a deal to me, unless of course you want to give me named variables in the signature besides just auto shifting off self, e.g. method foo ( $bar ) { return $self->{foo} if $bar }

Better Exceptions with Exception::Base

So I've done some complaining and explaining about what I'd like to see in regards to Exceptions in Perl. I Mostly explained what I wanted for catching Exceptions, and a little on throwing Exception objects, but not really how those objects should behave. I've looked at and tried various exception modules, including croak, confess, and Throwable. I basically spent time one day reading the manuals of most of the exception objects on CPAN. Most of them didn't allow me to easily for the exceptions I needed (meaning they required more work than I thought I should do for one or two exceptions ). Among these modules I found Exception::Base, which appears to do everything I need (though I still wish for something like it in core, with shinier syntax). Of course wee need to be able to throw simple Exceptions, preferably ones that can stringify, and are easily matched in a switch or if statement. Exception::Base can do all of that, and it even boolifies to true. A really big thing I wanted was a class I could easily add attributes to without writing a whole new package/pm and subclassing it there. I wanted this because I really wanted to be able to have 2 kinds of messages, one for programmers, and one for users, but truthfully I only had one class where I needed this flexibility (at the time). It is also occasionally useful to have attributes that describe something, e.g. would be really useful in moose attribute exceptions, to be able to have an object where you could catch the exception and get the attribute name without parsing. Fortunately Exception::Base can do this too. You'll notice if you run this script that in the warn, both usermsg and logmsg are printed because they are both string_attributes. You'll also notice that attr isn't printed at all, but that we can look at it directly to make other decisions. Exception::Base has other features such as setting the verbosity so you can go from a croak level message to a full stacktrace. It also allows you to ignore_packages so that the exception does not appear to be thrown from that package. This is similar to @CARP_NOT.

Perl Core Syntax Wishlist: die should die

I hate die it is, in my humble opinion, one of the worst parts of perl. I really wish it would be deprecated, and removed, or at least replaced with something that would tell you were the code that was die-ing was being called. Replace its implementation with that of Carp's croak or confess and I'd be happy. Better yet, let's just get real exception support and deprecate die (even if that's never removed deprecation just make that real big on its pod). If you're using die please stop and use Carp, anyone using your module will thank (and by thank I mean not curse) you later.

Perl Core Syntax Wishlist: an Exception Stack

I have come to wish many things were part of Perl syntax that are not, and no using external modules is not enough for me. I think it's time Perl got the features as part of the language itself (and yes I suppose I could settle for feature.pm, and no I'm probably not going to write them myself, I'm not smart enough yet). The first of these is a proper exception stack. I want to be able to write: I think we need throw, try, catch, and finally keywords. And no I don't think it makes sense to have Object->throw. In fact I think this Original Perl 6 Syntax Proposal reads like just what we need in Perl 5. Unfortunately I think this is what we are getting in Perl 6, which IMO is not as nice.

Adventures with SOAP using Perl: Part 2 ( SOAP::Data::Builder )

Start by reading the first 2 parts :

1. Part 0 Prelude (setup server.pl)
2. Part 1 SOAP::Lite

SOAP::Data::Builder is simply a wrapper around SOAP::Data and SOAP::Serializer for SOAP::Lite. I used it because it made my life easier building nested complicated SOAP objects. However for Part 2 I will simply be showing how to use it to do the same code as Part 1. Unfortunately since Part 1 is so Simple this actually makes SOAP::Data::Builder more complex than SOAP::Lite would be for this. In a future installment I will attempt to show more complex examples, but I will explain them less. Now let's take a look at the code.

#!/usr/bin/perl
use 5.010;
use strict;
use warnings;
use SOAP::Lite +trace => [ 'debug' ];
use SOAP::Data::Builder;

my $req = SOAP::Lite->new(
    readable   => 1,
    proxy      => 'http://localhost:8877',
    ns         => 'http://namesservice.thomas_bayer.com/',
);

my $sb = SOAP::Data::Builder->new;

$sb->add_elem(
    name  => 'name',
    value => 'Mark',
);

my $res = $req->getNameInfo( $sb->to_soap_data );

say '-' x 3;

unless ( defined $res->fault ) {
    say scalar $res->valueof('//country');
} else {
    say $res->fault->{faultstring};
    say $res->fault->{detail}{error};
}

As you can see this is much the same as the final code in Part 1, and if you run it, it does exactly the same thing. The first difference you'll notice is the self explanatory creation of the SOAP::Data::Builder object. After that comes the ->add_elem method call, which will simply create an element with a given element name and a value for that element.

Once you've finished adding elements to your $sb object, then you can call pass $sb->to_soap_data to the method that you're calling on SOAP::Lite, this generates the structure that SOAP::Lite needs to be able to make your request.

The only thing I didn't like about SOAP::Data::Builder is if you don't pass the right parameters to ->add_elem it will not croak or error in any way that will really tell you what went wrong. Simple patches to this can fix it.

Simple scripting CLI with Expect.pm

Expect is use primarily for sending Input to Command Line programs that Prompt and wait for input. For starters you'll need an executable script and I've pasted one that I got elsewhere for a demo.

#!/usr/bin/env perl
# slightly modified from http://www.tizag.com/perlT/perluserinput.php
use 5.014;;
use warnings;


print "How old are you?";
my $age = <>;

print "What is your favorite color?";
my $color = <>;

say "You are $age, and your favorite color is $color.";

It's trivial, you should probably run it just to see what it does. Now take a look at our Expect Script.

Although you can read the comments yourself, let's go over it anyways. ->spawn takes a list of arguments, with the first being a command, and the rest being any command line arguments, or options passed to said script. I could have easily just put the command in spawn, but I though I'd show a more dynamic example. After that you'll notice I've hard coded 2 sets of params that could possibly be fed into the program. I also have created some 1 at a time iterators, this simply makes it easier to iterate them in expect. Now onto the Expect object itself.

If you've read the documentation you'll notice there are 3 debug settings, with 3 being the most verbose, and some verbose settings. None of this appeared to be as verbose as I could get, however turning on ->exp_internal(1); printed everything I needed to understand expect and debug what I was doing wrong. The ->spawn method of course forks and execs our command, the output of which then gets iterated by ->expect.

The first argument to ->expect is the timeout, which is measured in seconds. The timeout is how long expect will wait for the output to match one of the regexes. You could set it to undef if you want to wait forever, or if it's just a program that has some startup time before it prompts set it to a few seconds. The timeout will (by default) be used again in between each prompt.

After timeout you can specify a list of array refs, each of which has 2 elements. The first is a regular expression that will allow you to tell expect how to recognize a prompt. The second is a coderef which allows you to tell expect what to do if it's corresponding match is hit. Both regexes will be run against every line, checking to see if they match.

Inside of our coderef we can use ->send to send input to the prompt. Remember send doesn't automatically press enter, so be sure to add your newlines. exp_continue means after match continue using this ->expect call to try to match the next line.

When running the script you'll notice the second regex never matches, but if reading the 'internal' output you'll see that it is attempting to.

The last line is a call to ->before which in this case, not very intuitively prints everything that was in the pty after the last match, or more appropriately, before the program exited. You'll also notice that it collects the thing that you sent to the last match.

That's the basics of what I've discovered, I'm not sure I fully understand how it all works yet (which means my explanations might not be 100% correct), but perhaps me writing this will help someone else get started with Expect. Happy Hacking!

Adventures with SOAP using Perl: Part 1 ( SOAP::Lite )

The most prevalent of SOAP libraries for Perl is SOAP::Lite it is the oldest and most documented. Though for all of its documentation it can be quite painful to figure out how to use it.

First make sure you've read Part 0 to set up the server. Once that's done let's look at the most simple way to interface with this server.

In our first example we need to send a request to getCountries, which is a method provided by the API. You can determine this by reading namesservice.wsdl and looking for the operations to see what's available. Essentially this means we need to send SOAP request with a Body of <getCountries />. First we need to import SOAP::Lite.
If you look at our SOAP::Lite import you'll notice that were are passing the arguments +trace => [ 'debug' ]. There are various levels and options for +trace, but this turns on full debug printing which will be sent to stderr. You don't normally want to have debug running in production code, but it will be useful to illustrate our examples and the request they send and receive.

Now let's look at creating an actual SOAP::Lite request object. The first option we pass in is readable => 1, adds whitespace to the request sent so that it's easier to read when you're looking at the debug output, you should not enable this in production, as it makes the request bigger, and I believe it is not considered correct SOAP as I've been told something about extra whitespace in SOAP being considered invalid. The second option is proxy => 'http://localhost:8877' This specifies the hostname and port that the HTTP request is sent to. ns => 'http://namesservice.thomas_bayer.com/' is the namespace, which you can find by looking for namespace in the namesservice.wsdl.

Now we need to actually create and send an actual request. For this trivial request we simply need to call the method that we need on the remote server and then return the object. You can see that SOAP::Lite is generating a namespace for your request to use with the XML <namesp1:getCountries xsi:nil="true">, which is just fine in this case.

Of course we want to do something with our response. Please note that I've modified the code to use 5.10, but if you want to use print instead of say this code will work fine on 5.6 and up. valueof, which is documented in SOAP::SOM, returns the first element in scalar context, and an array in array context. So in my code I've shown both. The syntax used in the parameters to valueof is XPath, so an even simpler way to call it in this case would be $res->valueof(//country); and it would do the same thing with this XML.

Next let's look at the getNameInfo method, it's a bit more complex so let's look at the XML in the XSD. Here's the snippet that is really important. This means that we need to send a request with a body that looks like ( note: you can look at the sample data in MyExampleData.pm for other names. ) Set let's take a stab at writing some Perl. There are some important differences to note from our previous script. You'll notice that I call ->getNameInfo() directly on the request object, instead of passing it as a parameter to ->call. This functions basically the same as call and it will end up making the first tag inside of body. We could have doen this in our first example as $req->getCountries; and that would have been it. Now that we've covered the slight differences in calls, let's go over the completely new things.

SOAP::Data objects are used to create any further data structures. Obviously the hash key of name defines the element name, and value defines what you want to go into it, here I have hardcoded "Mark".


If you run this code you'll notice that it returns a faultstring (among other fault properties) "operation getNameInfo for SOAP11 called with invalid data", and details the error as "element `c-gensym3' not processed at {http://namesservice.thomas_bayer.com/}getNameInfo". Now go back and look at the request, you'll see a c-gensym3 element, where did that come from? Well, SOAP::Lite will generate elements for anonymous elements but we can fix this.

The only difference between this and the previous code is that we aren't putting a \ in front of SOAP::Data. I wrote it the first way because I had seen examples of that all over the place, and could not find a solution to getting rid of the gensyms until I asked this question on stackoverflow.

 Unfortunately this is the most complex example that our server API has implemented. As an exercise to the reader I suggest Implementing a request for the method getNamesInCountry, which is no more complex but available.

Adventures with SOAP using Perl: Part 0 ( prelude )

This is a prelude to a series on working with SOAP Requests using Perl. For the past 3 months I have been working on a Perl API for CyberSource's Simple Order API which uses SOAP (I should note, that although I believe most of the API is now stable some area's still need work, and thus I don't expect it to reach 1.0.0 anytime soon).

First I used SOAP::Lite to do my requests, but I found it confusing to construct the requests that I needed to make. I even discovered a bug that lead to the current ( 0.714 ) release of SOAP::Lite.

Next I started using SOAP::Data::Builder to make it easier to build my SOAP::Lite requests. This was good, but frustrating that I had to add my data in a specific order.

Finally I came upon XML::Compile::SOAP. A glance at it's API which used a hash to build requests seemed much better. however, it took me a few weeks and some help from Mark Overmeer (the author) and an update to XML::Compile::SOAP::WSS to get it to work.

If you're planning on starting a new project that requires SOAP I definitely recommend using XML::Compile::SOAP if you have a .wsdl and a .xsd to work with.

I will be covering all 3 of these methods in Parts 1, 2 and 3 of the series.

To get started we'll need a SOAP Server since I haven't been able to find any reliable public services. To do this you can install XML::Compile::SOAP::Daemon (which probably could use some Plack/PSGI love ). You'll want to grab a copy of the namesservice example that is in XML::Compile::SOAP::Daemon, I have provided a patched copy in a gist . Once you've done that you can do perl server.pl . Now you should have a Server running on http://localhost:8877 which we can use for testing our client examples.

Please be advised, these tutorials will not be explain XML, XSD, WSDL, or SOAP, but simply the Perl interfaces.

UPDATE: Here is Part 1 ( SOAP::Lite )