Perl Core Syntax Wishlist: an Exception Stack

I have come to wish many things were part of Perl syntax that are not, and no using external modules is not enough for me. I think it's time Perl got the features as part of the language itself (and yes I suppose I could settle for feature.pm, and no I'm probably not going to write them myself, I'm not smart enough yet). The first of these is a proper exception stack. I want to be able to write: I think we need throw, try, catch, and finally keywords. And no I don't think it makes sense to have Object->throw. In fact I think this Original Perl 6 Syntax Proposal reads like just what we need in Perl 5. Unfortunately I think this is what we are getting in Perl 6, which IMO is not as nice.

Adventures with SOAP using Perl: Part 2 ( SOAP::Data::Builder )

Start by reading the first 2 parts :

1. Part 0 Prelude (setup server.pl)
2. Part 1 SOAP::Lite

SOAP::Data::Builder is simply a wrapper around SOAP::Data and SOAP::Serializer for SOAP::Lite. I used it because it made my life easier building nested complicated SOAP objects. However for Part 2 I will simply be showing how to use it to do the same code as Part 1. Unfortunately since Part 1 is so Simple this actually makes SOAP::Data::Builder more complex than SOAP::Lite would be for this. In a future installment I will attempt to show more complex examples, but I will explain them less. Now let's take a look at the code.

#!/usr/bin/perl
use 5.010;
use strict;
use warnings;
use SOAP::Lite +trace => [ 'debug' ];
use SOAP::Data::Builder;

my $req = SOAP::Lite->new(
    readable   => 1,
    proxy      => 'http://localhost:8877',
    ns         => 'http://namesservice.thomas_bayer.com/',
);

my $sb = SOAP::Data::Builder->new;

$sb->add_elem(
    name  => 'name',
    value => 'Mark',
);

my $res = $req->getNameInfo( $sb->to_soap_data );

say '-' x 3;

unless ( defined $res->fault ) {
    say scalar $res->valueof('//country');
} else {
    say $res->fault->{faultstring};
    say $res->fault->{detail}{error};
}

As you can see this is much the same as the final code in Part 1, and if you run it, it does exactly the same thing. The first difference you'll notice is the self explanatory creation of the SOAP::Data::Builder object. After that comes the ->add_elem method call, which will simply create an element with a given element name and a value for that element.

Once you've finished adding elements to your $sb object, then you can call pass $sb->to_soap_data to the method that you're calling on SOAP::Lite, this generates the structure that SOAP::Lite needs to be able to make your request.

The only thing I didn't like about SOAP::Data::Builder is if you don't pass the right parameters to ->add_elem it will not croak or error in any way that will really tell you what went wrong. Simple patches to this can fix it.

Simple scripting CLI with Expect.pm

Expect is use primarily for sending Input to Command Line programs that Prompt and wait for input. For starters you'll need an executable script and I've pasted one that I got elsewhere for a demo.

#!/usr/bin/env perl
# slightly modified from http://www.tizag.com/perlT/perluserinput.php
use 5.014;;
use warnings;


print "How old are you?";
my $age = <>;

print "What is your favorite color?";
my $color = <>;

say "You are $age, and your favorite color is $color.";

It's trivial, you should probably run it just to see what it does. Now take a look at our Expect Script.

Although you can read the comments yourself, let's go over it anyways. ->spawn takes a list of arguments, with the first being a command, and the rest being any command line arguments, or options passed to said script. I could have easily just put the command in spawn, but I though I'd show a more dynamic example. After that you'll notice I've hard coded 2 sets of params that could possibly be fed into the program. I also have created some 1 at a time iterators, this simply makes it easier to iterate them in expect. Now onto the Expect object itself.

If you've read the documentation you'll notice there are 3 debug settings, with 3 being the most verbose, and some verbose settings. None of this appeared to be as verbose as I could get, however turning on ->exp_internal(1); printed everything I needed to understand expect and debug what I was doing wrong. The ->spawn method of course forks and execs our command, the output of which then gets iterated by ->expect.

The first argument to ->expect is the timeout, which is measured in seconds. The timeout is how long expect will wait for the output to match one of the regexes. You could set it to undef if you want to wait forever, or if it's just a program that has some startup time before it prompts set it to a few seconds. The timeout will (by default) be used again in between each prompt.

After timeout you can specify a list of array refs, each of which has 2 elements. The first is a regular expression that will allow you to tell expect how to recognize a prompt. The second is a coderef which allows you to tell expect what to do if it's corresponding match is hit. Both regexes will be run against every line, checking to see if they match.

Inside of our coderef we can use ->send to send input to the prompt. Remember send doesn't automatically press enter, so be sure to add your newlines. exp_continue means after match continue using this ->expect call to try to match the next line.

When running the script you'll notice the second regex never matches, but if reading the 'internal' output you'll see that it is attempting to.

The last line is a call to ->before which in this case, not very intuitively prints everything that was in the pty after the last match, or more appropriately, before the program exited. You'll also notice that it collects the thing that you sent to the last match.

That's the basics of what I've discovered, I'm not sure I fully understand how it all works yet (which means my explanations might not be 100% correct), but perhaps me writing this will help someone else get started with Expect. Happy Hacking!

Adventures with SOAP using Perl: Part 1 ( SOAP::Lite )

The most prevalent of SOAP libraries for Perl is SOAP::Lite it is the oldest and most documented. Though for all of its documentation it can be quite painful to figure out how to use it.

First make sure you've read Part 0 to set up the server. Once that's done let's look at the most simple way to interface with this server.

In our first example we need to send a request to getCountries, which is a method provided by the API. You can determine this by reading namesservice.wsdl and looking for the operations to see what's available. Essentially this means we need to send SOAP request with a Body of <getCountries />. First we need to import SOAP::Lite.
If you look at our SOAP::Lite import you'll notice that were are passing the arguments +trace => [ 'debug' ]. There are various levels and options for +trace, but this turns on full debug printing which will be sent to stderr. You don't normally want to have debug running in production code, but it will be useful to illustrate our examples and the request they send and receive.

Now let's look at creating an actual SOAP::Lite request object. The first option we pass in is readable => 1, adds whitespace to the request sent so that it's easier to read when you're looking at the debug output, you should not enable this in production, as it makes the request bigger, and I believe it is not considered correct SOAP as I've been told something about extra whitespace in SOAP being considered invalid. The second option is proxy => 'http://localhost:8877' This specifies the hostname and port that the HTTP request is sent to. ns => 'http://namesservice.thomas_bayer.com/' is the namespace, which you can find by looking for namespace in the namesservice.wsdl.

Now we need to actually create and send an actual request. For this trivial request we simply need to call the method that we need on the remote server and then return the object. You can see that SOAP::Lite is generating a namespace for your request to use with the XML <namesp1:getCountries xsi:nil="true">, which is just fine in this case.

Of course we want to do something with our response. Please note that I've modified the code to use 5.10, but if you want to use print instead of say this code will work fine on 5.6 and up. valueof, which is documented in SOAP::SOM, returns the first element in scalar context, and an array in array context. So in my code I've shown both. The syntax used in the parameters to valueof is XPath, so an even simpler way to call it in this case would be $res->valueof(//country); and it would do the same thing with this XML.

Next let's look at the getNameInfo method, it's a bit more complex so let's look at the XML in the XSD. Here's the snippet that is really important. This means that we need to send a request with a body that looks like ( note: you can look at the sample data in MyExampleData.pm for other names. ) Set let's take a stab at writing some Perl. There are some important differences to note from our previous script. You'll notice that I call ->getNameInfo() directly on the request object, instead of passing it as a parameter to ->call. This functions basically the same as call and it will end up making the first tag inside of body. We could have doen this in our first example as $req->getCountries; and that would have been it. Now that we've covered the slight differences in calls, let's go over the completely new things.

SOAP::Data objects are used to create any further data structures. Obviously the hash key of name defines the element name, and value defines what you want to go into it, here I have hardcoded "Mark".


If you run this code you'll notice that it returns a faultstring (among other fault properties) "operation getNameInfo for SOAP11 called with invalid data", and details the error as "element `c-gensym3' not processed at {http://namesservice.thomas_bayer.com/}getNameInfo". Now go back and look at the request, you'll see a c-gensym3 element, where did that come from? Well, SOAP::Lite will generate elements for anonymous elements but we can fix this.

The only difference between this and the previous code is that we aren't putting a \ in front of SOAP::Data. I wrote it the first way because I had seen examples of that all over the place, and could not find a solution to getting rid of the gensyms until I asked this question on stackoverflow.

 Unfortunately this is the most complex example that our server API has implemented. As an exercise to the reader I suggest Implementing a request for the method getNamesInCountry, which is no more complex but available.

Adventures with SOAP using Perl: Part 0 ( prelude )

This is a prelude to a series on working with SOAP Requests using Perl. For the past 3 months I have been working on a Perl API for CyberSource's Simple Order API which uses SOAP (I should note, that although I believe most of the API is now stable some area's still need work, and thus I don't expect it to reach 1.0.0 anytime soon).

First I used SOAP::Lite to do my requests, but I found it confusing to construct the requests that I needed to make. I even discovered a bug that lead to the current ( 0.714 ) release of SOAP::Lite.

Next I started using SOAP::Data::Builder to make it easier to build my SOAP::Lite requests. This was good, but frustrating that I had to add my data in a specific order.

Finally I came upon XML::Compile::SOAP. A glance at it's API which used a hash to build requests seemed much better. however, it took me a few weeks and some help from Mark Overmeer (the author) and an update to XML::Compile::SOAP::WSS to get it to work.

If you're planning on starting a new project that requires SOAP I definitely recommend using XML::Compile::SOAP if you have a .wsdl and a .xsd to work with.

I will be covering all 3 of these methods in Parts 1, 2 and 3 of the series.

To get started we'll need a SOAP Server since I haven't been able to find any reliable public services. To do this you can install XML::Compile::SOAP::Daemon (which probably could use some Plack/PSGI love ). You'll want to grab a copy of the namesservice example that is in XML::Compile::SOAP::Daemon, I have provided a patched copy in a gist . Once you've done that you can do perl server.pl . Now you should have a Server running on http://localhost:8877 which we can use for testing our client examples.

Please be advised, these tutorials will not be explain XML, XSD, WSDL, or SOAP, but simply the Perl interfaces.

UPDATE: Here is Part 1 ( SOAP::Lite )

Experience with having a non Dist::Zilla user contribute to a Dist::Zilla project (it's not hard for them or me)

I've heard many times that Dist::Zilla makes it harder for people to contribute to the project. This is not true, it is either unfortunately either ignorance or FUD (much like saying Linux is harder to use than Windows). Truly, there are things that some dzil users do that can make it harder, but it doesn't have to be that way. Michael Schwern recently contributed to one of my dzil projects without ever realizing I was using dzil, until I told him. He more recently stated on twitter, "While your solution works, it seems like it makes more work for you to shield contributors from dzil". This was true in this case because I wasn't sure how to effectively move a series of multiple patches, I now know it's easy to do with git. Here's how you can allow contributors to contribute to your dzil project without causing you or them undo pain.

1. Don't use anything that changes the line numbers of your source

   Examples: don't use DZP::Prepender. Use modules like OurPkgVersion to insert VERSION, and make sure your # ABSTRACT and any pod is below the 1; at the end of your module. This will keep the line numbers of errors in your code from being different from the final build. It's still ok to use PodWeaver as it will save you way more time than it'll hurt, so long as you follow the rules about pod being at the bottom





2. Use a plugin that commits builds to your source control software

   I personally recommend DZP::Git::CommitBuild









3. Make your build branch your default branch

   I'm not sure how to do this with just git, but on github you can go to the admin section of your repository and change the default branch there. This makes it so that when someone clone's your repository the initial checkout is of your build branch. Your build branch shouldn't require dzil, it's the final build.

Now that you have your final build branch as the default branch anyone who wants to contribute can simply clone your repo and start hacking. Their are a couple of mistakes they could make, They could either change or add files that are maintained, or pruned by dzil. This did happen when Schwern sent me patches, one of them was the addition of a .gitignore, which I already had, but is being pruned out of the build branch. I can also see it happening to meta's and makefiles. These patches can simply be rejected as unnecessary, dzil-ified, or if they are truly a bug, then they can be reported and fixed upstream.

So what if someone sends you a pull request from build/master? well if it's just one or two patches, you add their remote, and do a git remote update and then you can git log remote build/master to find the sha1 of this patch. Now that you have the sha1 all you have to do is git cherry-pick [sha1] and it should apply. If there are any conflicts you may have to resolve them with git mergetool. However, the only conflict with Schwern's patches for me was the .gitignore, all other patches applied without assistance and applied correcly, surprisingly even the pod patch applied without issue.

If you have more patches than is comfortable with git cherry-pick then you need git rebase. The command you want is git rebase -i --onto master [sha1 before first sha1 in series] [tip of remote branch checkout]. So in my case git rebase -i --onto master 3f1e3748 schwern. What this appears to do is ends up rewriting my local schwern/build/master checkout and removes all the build commits, and then applies the patches on top. This means I can now do git merge schwern from the master branch, and all of his patches that I want will be successfully merged. For more on this strategy you may want to read this stackoverflow question.

given this is not quite as easy as a git pull that's a fast forward, but reality is it's not that hard once you know how to do the rebase and how it works. Of course this isn't ideal for constant contributors, those should simply learn to use dzil, but for the random contributor it should be ok.

My solution to not using PluginBundle:AUTHOR for dzil is git

I am now tired of updating my dist.ini's for my Dist::Zilla projects. For many people this is when they start using a PluginBundle with their authorname. I discussed why you shouldn't do this a year ago. Now that I'm tired of managing my dist.ini's on an individual basis I'm going to show you how I'm going to solve the same problem everyone else is, which is getting tired of updating your dzil configuration for all of your projects. I'm using git to do it. You probably haven't considered this, or am thinking I'm wrong because you believe that git can't merge branches without a common history, which is not true. I did it with Regen2, Funtoo, Portage, and Sunrise, which have way more files than any perl repo, including Perl.


First let's talk about the advantages and disadvantages of doing this with git.

I'll start with the disadvantages. You'll have another repository to manage. Git isn't completely automatic, you'll have to remember to merge your changes. You'll also have to add the repository to your existing repositories. You will have to resolve merge conflicts at least once, and probably occasionally more, though most should be fast-forwards.

The advantages are... now the changes to your dist.ini are getting recorded in your history. You can now have a master dist.ini, but remove items without filters in your individual modules. You can share even more configuration as merged differences allow you to maintain differences in downstream commits. It doesn't automatically update all of your modules. Wait didn't I list that last one in disadvantages? Yep, it's an advantage because what if you update your PluginBundle and that update breaks one of your modules, but you don't know it because you haven't worked on that module in a while. You can use this for more than just dist.ini.

Ok so the first thing you you want to do is create your master dzil repo. This is not a git or dzil tutorial so go do that. I did it by creating a dzil new project and removing the files I couldn't use and making a few tweaks and amending all of those changes to the initial commit. Check out my repo for inspiration. Remember the directory structure has to be the same as a dzil repo structure for any common files.

Now add a remote from your new repo to an existing dzil project. Next you need to merge the branch from the remote into your project. The first time you do this you'll have to resolve conflicts. If they're like mine then they'll be easy and obvious. Once that's done future changes will be fast-forwards unless you change something in your perl module repo, then you might end up with a simple merge conflict. One thing to remember NEVER EVER try to merge from your module repo to your dzil repo, it will cause you extreme pain in the future. cherry-picking that direction is possible but not merging.

After that, all that is left to do is run your dzil tests and fix any breakage in your module. Happy merging.

Regen2 accounts transferred

I killed regen2 about 2 years ago. Today I was contacted by Rafal Kupiec asking about the project. Since it's dead he asked if he could take over the project. I have transfered the github account, and freenode channel #regen2 to him. The google groups are remaining archived. At some point in the future there should be a site at asiotec.org I have no idea what he plans on doing with it.

Test::Version under maintenance again

About 4 months ago russoz (Alexei Znamensky) sent me a message about quoting the versions generated by Dist::Zilla::Plugin::OurPkgVersion. I was a bit skeptical at first about needing it, and the suggestion seemed to have more to do with style anyways. But after digging a little I found to my horror that OurPkgVersion was generating completely invalid multipart version strings, e.g:

our VERSION = 0.1.0;

Which doesn't work. I was shocked because I was using Test::HasVersion. Well all Test::HasVersion does it checks to see that you have a VERSION defined. My code above will pass a defined test, but you certainly won't get anything if you try to print it.

So I accepted Russoz's patch and fixed up my documentation. Still, I was disturbed that there weren't any Test modules that would have caught this error for me.

So, eventually, I began working on writing a Test module to do this task for me. First it was mostly research and learning which modules and methods I would need. One of the hardest parts was determining how best to extract the VERSION from the module. PPI seemed like an impossible task, probably most due to the fact that I don't understand it. I thought about require-ing modules, but many told me that could lead to a vulnerability in the test. Ultimately decided to use Module::Extract::VERSION which will work for everything except perl 5.12 style versions. Then I had the problem of learning how to write Tests for Test modules, which ultimately doherty (Mike Doherty) contributed some patches to get my first set of tests working.

Once the module was all but finished functionality wise I had to decide on the name, after discussing it on #distzilla it was decided that it might be a good idea to take over an existing module. Test::Version seemed like the most obvious candidate because it has the most obvious namespace for what is trying to be done. So I spent the next few weeks trying to contact particle (Jerry Gay) to get maintainer-ship. Once I did manage to contact him it wasn't an issue to get control of the Module.

So I released a new version of the old Test::Version 0.03 announcing new maintainership, on the off chance anyone was using it in the darkpan, since I'd already searched the CPAN archives for any usage and no current module was. I then released a TRIAL version for testing. The Trial version ended up having several failing tests by cpan tester reports, I thought it was caused by Test::Builder::Tester whitespace sensitivity (and maybe some of it was). rjbs (Ricardo Signes) suggested Test::Tester and I released yet another version with failing tests. This time I analyzed about 5 of the test reports, and come to find out the module seems to run the tests on the perl modules in no particular order. I assume that somewhere there is some speed optimization which results in this behavior. So I had to find a way to write the tests which could handle the fact that I couldn't depend on the order that files were being tested. Ultimately worked that kink out as well in a way that appears to be satisfactory.

So with version 0.07 we now appear to have a working Test::Version. That checks to see if modules have versions, and that they are valid. It's also capable of checking an entire dist. The docs describe use, and if you use Dist::Zilla you can check out Dist::Zilla::Plugin::Test::Version or use Dist::Zilla::Plugin::TestingMania which will be including it.

Going forward I plan to take version 0.07 if it turns out to have no serious bugs, and release version 1.0.0 as it stands. After that I have a few features releases in mind. Including supporting is_strict if you'd like to make sure that your VERSION passes those tests as well. I'd also like to pull in the functionality of Test::GreaterVersion and Test::ConsistentVersion. I'm open to other suggestions as well.

I hope people can make some good use out of this module to avoid silly mistakes like I made.

Happy Testing!

New beginnings

Since I truly last posted a lot has changed. I moved to Houston, TX, USA (from MI) for a position as a Linux System Administrator. As of this week I've been moved to the programming department in my company, coding Perl. I'm also an elected moderator on StackExchange's Unix & Linux. Of course there's always updates to my CPAN dists.