Generating indexed items

This is a post about generating new items in a pipe where those items are sequentially indexed based on a given number.

What would we use this for?
One reason that comes to mind is where the data is spread over a number of web pages. So we go to the first page and there are links to the next 3 pages and a link to the last page. What we want to do is fetch all of the pages. The link to the last page could look like
http://www.somedomain.co.uk/greatstuff?page=17
So we have the number of the last page and need to generate URLs for pages 2 to 17 (we already have page 1).

Background
Paul Donnelly in a Yahoo! Pipes blog post has already shown us how to index a set of existing items.
http://blog.pipes.yahoo.net/2009/06/01/using-yql-execute-to-power-the-pipes-webservice-module/
Look in the last paragraph of that post for the index example.
So the main task that is left is to create the required number of items.

Generating n items
The pipe starts by putting a numeric value into an element "number" and then reducing that by 2. That’s because Paul’s indexing starts with 0 and we want the indexing to start with 2 (in our example usage we already have page 1). The next step is to use a Regex module to create a new element "num" that includes our number.

Imagine an item that had an element "ting" which was an array where the last cell was cell 15. How would we create that array?
This is step 1.

We’ve now created an element "ting" that has a sub-element "15" that has a value "fred". The element name "ting" is not significant. Neither is the value "fred". The important thing to notice here is that we have not given item.num a new value. What we have done is to assign a value to the contents of item.num. Let’s have a look at this when we assign the value of the Loop instead of emitting it.

We can see that the element "num" still has the value "ting.15", but in loop:itembuilder we have an element "ting" with a sub-element "15" which has a value "fred".
So the next step is to take this element "ting" and a sub-element "15" and create a full array.

From there it’s fairly simple to pass the array through Paul Donnelly’s routine and adjust the numbering (as in my pipe). And from there I’ll leave it to your ingenuity to use the "yindex" element to create URLs.

Advertisements

Tags:

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: