Concatenation revisited

Now that Paul Donnelly has shown us the way, let’s look at another way to concatenate elements from items that uses YQL and the Web Service module.

This pipe starts off in the same way as the pipe in my previous post on concatenation, with a URL Input, an Item Builder and a Fetch Feed in a Loop.

Here we use Rename to make a copy of the items (in item.stuff) to an element yql_array. That’s a specific name that will be used by the YQL web service. On the right is a URL Builder with the elements:

  • Base –
  • Path – v1/public/yql
  • Parameters
    • q – use as array_concat; select * from array_concat where data = @data and delim=@delim and ele=@ele
      This is the YQL statement at the heart of this method. The use clause links to a YQL open data table with an  Execute clause in it, and gives that table a name of array_concat. The select clause is going to pass values for “data”, “delim” and “ele” to the open table. The values for these are specified using YQL variable substitution. It looks like the @data substitution is special, since we ourselves don’t supply a value for it.
    • format – json. That’s the format the Web Service uses.
    • diagnostics – If you’re having problems with a YQL statement you can set this to “true” and get diagnostic information.
    • delim – One of the @ variables that were used. The delim value is a separator for each of the elements we’re going to collect.
    • ele – the element name. In this case we’re concatenating title elements.

In the Web Service module the path is set to query.results.result.items.

The result of the concatenation is always returned in the yql_array_string element. The rest of the pipe then works on that string to produce the desired output.

This approach isn’t just limited to working on whole items as we can use it to concatenate array elements within an item. In this pipe, for instance, category values within an item are being concatenated. This is how the data looks to start with:

In the Rename module we have

and in the URL Builder we have

Update: 10th June. The ele parameter is now optional. This allows for arrays such as category.0, category.1 as well as category.0.title, category.1.title.



9 Responses to “Concatenation revisited”

  1. Carl-Johan Sveningsson Says:


    thanks for a great blog, and excuse me for commenting again, but this is more in context. You’re describing concatenation, but how hard would it be to “demultiplex” 4-groups of TR’s like in this screenshot, assuming I’ve got them as a feed of items now?

    Best regards

  2. Trejkaz Says:

    This is a lot of work… for something that Yahoo should have given us as a very basic operator.

    If only we could write scripts to supplement all the missing features in Pipes. 😦

  3. Trejkaz Says:

    Actually, I get a 400 error from the server every time I run my attempt. Even if I set diagnostics=true, no diagnostics are sent, which doesn’t help at all.

    My pipe:

    What is wrong with it? The structure looks right, the parameters look right. I copied most of them verbatim from this blog post.

  4. Trejkaz Says:

    Oh… never mind, I figured it out. Your blog post says:

    q – use as array_concat; select * from array_concat where data = @data and delim=@delim and ele=@ele

    But actually you need double quotes around that URL.

  5. Josh Says:

    This don’t work on V2 engine, can you fix it?

    • hapdaniel Says:

      Thanks for pointing that out. When I first looked it wasn’t working but now is (at least some of the time).
      I’m not sure where the problem might lie when it doesn’t run – it could a Pipes V2 issues, a Github issue or a YQL issue.

  6. rnx Says:

    Is your pipe still working? It seems to me that it doesn’t, or I don’t understand it at all. I am trying to concatenate the content of all the items of a feed into a single item (

    • hapdaniel Says:

      Sorry about that. I thought I had updated the example pipes. I have now. Both pipes needed a change to the path in the Web service module, which should read

  7. rnx Says:

    Thanks! I managed to make it work!

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: