Passing complex elements to sub-pipes

I have 2 warnings here. The first warning is that, as far as I know, this technique is completely unsupported. It became possible as a by-product of the change the Pipes team made to allow users to enter their own values into drop-down boxes. The second warning, and this is a general one for passing any data to a sub-pipe, is that there is a limit of about 2,000 characters that can be passed to a sub-pipe.

So what am I talking about? I’m calling an element "complex" if it has one or more sub-elements.

Sio let’s have a look at some pipes. In the main pipe I’m using 2 Item Builders and a Union. Normally there would be some sort of Fetch module instead.

I’m passing item.stuff to the sub-pipe.

Let’s start with the simplest possible sub-pipe for this situation.

Notice that there are no default or debug values in the Text Input. I haven’t the least idea what this complex element is going to look like to the Text Input module. This makes things tricky, because I’m just going to have to assume that I get things right in the sub-pipe and wait until I’m back in the main pipe to check if everything’s working as expected. I’ve used the element name "stuff" in the Item Builder. I could have used any name, but I’ve found that it makes things simpler to use the same name as that of the element being passed to the sub-pipe.

Back in the main pipe we can now have a look at the result of all this.

The output of the Loop has returned, within the item.result element, a copy of the original stuff element. So far so useless. So let’s set about trying to do something useful. Let’s say I want to filter items to permit only items where a stuff.category element contains "Robots". Here’s an attempt at this without trying to use a sub-pipe.

There’s a Split module, and then in the right-hand path I use a Sub-element module to get the category sub-elements as items, filter on my key word and produce a count of the accepted items.

In the left-hand path I use a Regex module to get the Count value from the right path and then filter the items based on that value. Now all this would be great if my mission was to accept all items if any of the categories for any of the items contained the key word, but that’s not the aim. The problem here is that the sub-element is working at the feed level, so on all the items at once, while I need something that will work on the category values for each item. The purpose of this charade is that I can compare how things would look if I used the sub-pipe approach.

The Split and Regex in the sub-pipe are just there as mechanisms for getting the count value out of the pipe, since the Count module cannot be directly connected to the Pipe Output module. In the main pipe there is a Filter module to filter where the value of item.result.count is greater than 0. As I hope you can see, if you keep the name of the element in the sub-pipe the same as the name of the element passed from the main pipe, it’s not too difficult to work out what is needed at the level of the main pipe and then apply that inside the sub-pipe.

If I hadn’t wanted to show a comparison between processing in the main and sub-pipes I would probably have not used a Count in the sub-pipe, but rather used a Truncate module, to 1, after the Filter and connected that to the Pipe Output. In the main pipe I would have set the Filter to permit items where result.content matched the regex "^(?=.)", that is, result.content existed at all (by containing a character).

Advertisements

Tags:

10 Responses to “Passing complex elements to sub-pipes”

  1. Carl-Johan Sveningsson Says:

    Wow, that’s some advanced stuff!

    Any idea how to go about chunking a list of undetermined length by groups of four like in this XHTML? http://twitpic.com/c46xp

    I’m really trying to find out what would be the best way around this, your comments would be extremely valuable.

    Cheers
    CJ

  2. hapdaniel Says:

    Carl-Johan,
    It would be better concerning both your questions if you posted on the Yahoo! Pipes message boards, or you can email me at hapdaniel_at_googlemail.com.
    Some more information about the background to your questions as well as links to some real live data would be useful.

  3. Johnny Baillargeaux Says:

    Hi,

    Thank you very much for that topic. I am trying to concatenate 2 items (0.content & 1.content) using this methodology but i am not able to.

    Any chance to look at my pipe (http://pipes.yahoo.com/pipes
    /pipe.edit?_id=Im8u_D9f3hGpVUtlPm7D0g) in order to give me your input ?

    It would be great if Yahoo could create a module in order to that…

    Thanks in advance,
    Johnny Baillargeaux (from france)

    • hapdaniel Says:

      Johnny,
      You are using the wrong technique for your pipe.
      After the Loop and String Builder in the left hand part of the pipe add a Fetch Data module in a Loop, and attach the right-hand URL Builder to the URL for the Fetch Data. You need to assign the first result and not all results. I expect that the element the result goes in will be item.loop:fetchdata.
      You can then attach the Loop and String Builder from the right-hand side and connect this after the Fetch Data.
      Change item.total to item.loop:fetchdata.total and so on, and assign the results to item.available.
      You will then have two elements, item.content and item.available, that you can join using modules such as String Builder or Regex.

  4. Pasquale Ondrey Says:

    By far the most concise and up to date information I found on this topic. Sure glad that I navigated to your page by accident. I’ll be subscribing to your feed so that I can get the latest updates. Appreciate all the information here

  5. Jason Says:

    Hi. im reading your article from the UK in IE8 but i can get the pictures to display. i know its been a while since the article was posted but it is availble to view anywhere else? thanks Jason

  6. Josh Harle Says:

    Hi hapdaniel,

    I’ve been trying various methods of getting multiple media:content nodes in my RSS items, but none have allowed an arbitrary number, and none have been at all elegant. I’m sure this is a problem you’re completely familiar with and have dealt with before.

    Any advice?

    Josh

  7. Elvis Lynah Says:

    It is nice that you blogged about this. I found you on google and I had been searching for info about this. Nice blog, thank you for the info.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: