[Note: this video only just updated, so quality will improve over time.]

As I’ve mentioned previously, an unfortunate reality of the current Twitter Search tool is the distinct – and short – lifespan for Tweets. After a matter of a few weeks content ceases to appear in the search results, effectively wiping them from the face of existence.

In my mind this is an absolute travesty. An enormous wealth of information is freely shared every day and is worth preserving for posterity – not only from the standpoint of access to links and resources, but also analysis of trends over time.

Therefore one of the pet projects I’ve been indulging in over the last year or so, Tweets in Perpetuity looks at how best to archive Twitter posts so they are searchable and easily referenced later. A number of people have asked how the system actually works, so I wanted to take the time to document it.

The basic building blocks of the system are:

• WordPress – http://wordpress.org
• FeedWordPress – http://wordpress.org/extend/plugins/feedwordpress/
• P2 Theme – http://wordpress.org/extend/themes/p2

In the case of P2 and FeedWordPress, both of these elements can now be installed directly from the WordPress Dashboard area. Please see the WordPress Docs area for information on Installing Plugins and Adding Themes.

Finally, a major shout-out goes to Jim Groom, whose work with P2 at UMW Blogs inspired this whole idea and yet I failed to acknowledge in the video.

Personally I’d prefer Twitter updates stay out of Buzz completely.  If I want to stay up to date with someone’s tweets, I’ll follow them on Twitter.  What we see happening in Buzz these days is the same fate that befell FriendFeed – the complete inundation of one system by updates from another service.

In FriendFeed’s case there is the option to filter certain types of updates and effectively keep tweets from overwhelming the experience.  Unfortunately no similar option exists in Buzz.  So in my case I’ve found myself unfollowing large numbers of people I would otherwise have included as contacts.

That said, many of these people weren’t engaging via Buzz anyway, so there was no real benefit from connecting with them there.

At this point I’m still looking at Buzz as an alternative to what I had been doing on Facebook (FriendFeed is ruled out because that’s now owned by Facebook).

Diaspora is definitely something I’m keeping my eye on as well. However with this service seemingly still only in the planning stages, it’s not a viable alternative at this point.

Fortunately for me some of the key people I want to interact with are starting to use Buzz for actual meaningful discussion, and not just re-mixing the same content that’s happening elsewhere.   What’s more, the relatively small networks that are beginning to develop are such that there isn’t a great deal of noise taking place – so some really insightful comments and discussions have occurred.

So please, think twice before you aggregate your tweets in Buzz.  Just because you’re able to doesn’t mean you should.

From their blog:

“Every public tweet, ever, since Twitter’s inception in March 2006, will be archived digitally at the Library of Congress. That’s a LOT of tweets, by the way: Twitter processes more than 50 million tweets every day, with the total numbering in the billions.”

What I’ve been unable to determine so far, is whether this information will be publicly searchable online.  I certainly hope it will be, but stranger things have happened.  It’s also unclear whether this applies only to Tweets that were shared publicly, or those that were restricted to only the user’s approved contacts.

The significance of this is that Tweets tend to fall off the radar relatively quickly, with Twitter’s native search functionality (at least at one stage) only returning search results from the last 2 weeks or so.  If you were hoping to reflect back on a backchannel discussion that occurred during a conference, review a collection of resources that was posted, or perhaps track the discussions of an open course – even if a hashtag had been used – you were relatively out of luck.

This is the reason I set up Tweets in Perpetuity, to “preserve the stream 140 characters at a time.”  The opportunity to participate in public discussion is incredibly valuable, the ability to reflect back on them later is perhaps even more so.

I’ve discussed how I set-up T.I.P before (“Tweets in Perpetuity: An experiment in syndication“), the point was that I needed a way to sensibly store and index a selection of Twitter streams for easy reference later, and Twitter wasn’t doing that.

My hope is the Library of Congress will establish something similar so the rest of the world can take advantage of the monolithic collection of data their acquisition represents.  What came from the public digitally should remain accessible to the public, digitally.

In the end I gave up. After many days of looking and searching – first for an easy solution and then progressively more and more complex ones as the week drew on – I ultimately realised I was developing an increasingly convoluted design merely to stay with a free public blogging service.

As it turned out this issue affects both WordPress.com blogs and Blogger/Blogspot blogs – there does not seem to be a solution in place for either platform. I have a sneaking suspicion this restriction is to combat spam abuse – because really it doesn’t arise from technical constraints. Services like Diigo have shown us how incredibly easy it is to set something like this up, the point is someone has chosen not to let us.

As proof of this, once I made the decision to go with a self-hosted installation of WordPress on my ISP, I had the solution up and running in a matter of minutes.

At a very basic level, all you need is a vanilla installation of WordPress (most commonly available through CPanel on your web host’s admin area), and a plugin like Twitter Tools, which is the plugin we ended up using.

Once these two elements have been installed it’s just a matter of configuring the Twitter Tool settings within the WordPress Dashboard area (using the Experimental Digest options), and you’re up and running.

What really sucks in all this is that the average Joe or Jane Blogger are unable to implement this sort of system because the business rules of the blogging providers are getting in the way.

The fundamental objective is to develop a process of capturing a day’s worth of Twitter posts, and auto-posting them in digest form to a Blogger/Blogspot blog.

The logic behind this is to help with a note-taking, record-keeping process that we are responsible for, in which we can quickly jot down points through-out the day via Twitter, and have an easy way to review and summarise them later.

You could of course suck in the RSS feed for an individual account, pass it through Posterous, and then autopost it to Blogger, however this would create a new post for every single Tweet, which is not what we’re after.

Yet despite this, for the life of me I cannot locate an easy way of implementing this system.

With WordPress the solution is quite straightforward – there are any number of plugins that can be integrated to the base blog software that will do all this for you.  Unfortunately with Blogger no such plugins seem to exist.  So what is exceptionally easy in one platform is increasingly becoming a complex system design process in the other.

Google Alerts

I’m was looking at Google Alerts as an option, however doesn’t seem to do what I need it to either.  While I can seem to get Alerts sending me daily email digests of posts “from:http://twitter.com/mbogle” the daily digest option is unavailable when you select “deliver to feed.”  You end up with the same itemised stream of updates that you started with.

Yahoo! Pipes
The other possibility I was considering was trying to set up a “pipe” using Yahoo! Pipes, and see if this might do the job, but having looked at the system I’m finding I’m just smart enough to design a solution – or indeed determine if it’s even possible – so the constraints of my technical skills have more or less scuttled that possibility.

More investigation required…
At this point I’m coming up empty-handed no matter which way I turn, so I’ll just have to keep looking.  Ultimately I find it amazing that no system exists yet for Blogger that will let you do this.

Not sure why that is, but if you’ve got tech skills it would seem to be a good opportunity.

If I can manage to figure out how on earth to do this I’ll post the solution here.  Otherwise if you happen to know a way I’ve overlooked PLEASE post a comment or trackback here and enlighten me.

Mike, out.

In many cases I have no other way to get in contact with people – no email addresses, no instant messaging contacts, no memorized blogs or wiki addresses.  I rely exclusively on access to a single service for much of this information.  In IT terms, there is no redundancy, no fail-over, no data back-up – and that’s very bad practice.

At one point some time ago I spent a week looking for an alternative to the service, and ultimately the conclusion I came to is there are many good services out there – or at least there were before Jaiku and Pounce both met their demise.  However the one crucial element each of them is missing is the human element – the people, the social framework, the network.  The source of information, input, expertise and knowledge; the people I have existing relationships with.

Changing services is one thing – that’s the easy part.  Taking your network with you is an entirely different matter.

Yet once again, Twitter is down, and I’m reminded of how dangerous it is to rely so much on a single service, when all services are destined to experience downtime from time to time.

So the question remains, what substitutes for Twitter?

Update: According to TechCrunch, the Twitter outage was due to the fact it was hacked.

Components

As indicated, the syndication framework I’ve set-up incorporates WordPress as the blogging platform, FeedWordPress for aggregation and syndication, and a WP Theme called P2, which is designed as “a group blog theme for short update messages.”  Add in Akismet for spam protection, and WordPress.com Stat’s to “tracks views, post/page views, referrers, and clicks” and that’s everything.

From a technical standpoint it’s a very, very basic WordPress framework, however as I’ve discovered it works amazingly well for the task I had set out to achieve.

I’ve christened the site “Tweets in Perpetuity” and it is available at: http://p2.techticker.net/

Syndication

In terms of setting up the syndication elements, I’m currently experimenting with a couple of use cases – the feed for my Twitter updates, as well as the search results for #CCK09 and #ECI831.  The latter two feeds were important to test out, because I wasn’t sure exactly how WordPress would handle the fact the feed contained contributions from different authors.

As it turns out, the FeedWordPress settings give you control over a variety of different elements – including Authors, Posts, Categories and Tags.

For the Author setting, I’ve configured FeedWordPress to create a new WordPress account for Authors who haven’t been syndicated before.  This means it’s easier to filter out the contributions of different people so they’re not all lumped into one general category.  From the standpoint of attribution and citing sources this is very important – I don’t own the content and do not to wish have it appear as such.

Following on this logic, I’ve configured the permalinks to point back to the source on Twitter, rather than the local instance syndicated in the WordPress blog.

Categorising the Feeds

Due to the fact I currently have 3 different feed sources coming into the blog, it made sense to provide additional organisational structures  to differentiate one stream from the others.  To accommodate this I’ve set FeedWordPress to automatically assign each post in a feed to a specific category as soon as it’s syndicated and have used a term that is clearly related to the feed – in this case @mbogle, #cck09 and #eci831 respectively.

So now, using the category links in the right menu, you’re able to quickly filter the updates to one stream.

Search Indexing

One benefit of syndicating through WordPress that I’d failed to remember is the fact that, once the content has been pulled in and syndicated, it is also indexed and therefore searchable.  So now you’re able to take advantage of WordPress’s awesome full-text indexing and locate information far more easily that you can with the native Twitter search options.

Comment Fracturing

One factor that I’m still grappling with is how to mitigate comment fracturing, and ensure that any discussion that takes place on the WordPress blog is channeled back into Twitter and made available to the author of the content.

One of the settings in FeedWordPress concerns whether to allow commenting or not.  In the case of syndicated blogs it’s best to deactivate commenting and refer people back to the point of origin to engage in discussion there.

At the moment I have the commenting option activated, and people are allowed to post replies on the blog – however I’m in two minds about whether that should remain the case or not.

I spent quite a while trying to locate a plugin for WordPress that would enable people posting comments to also sent their comments to Twitter, but the one that looked promising does not appear compatible with the P2 theme I’m using.

The other option is to try incorporating a commenting framework such as Disqus, which includes this option as well – but again, I’m not sure whether P2 would be compatible or not.

Implications of Copyright on Syndication

Much more significant a topic than comment fracturing though is that of copyright, and the implications of copyright law on syndication frameworks such as this one.  I am not really all that cognizant of what the legal implications are of syndicating other people’s Tweets, and need to investigate this further.

For example, I’m uncertain about where fair use comes into play versus the need to seek approval prior to replication or reproduction of content.

If it is true that the moment a piece of content is created, it’s immediately copyrighted by the author, then theoretically I am violating the copyright of everyone I’ve syndicated so far who has not explicitly released their material under an open license.

For the moment what I’m going to do is include a note on the site providing my email address and requesting that anyone who does not want their content reproduced to contact me and I will ensure their material is removed from the site and not syndicated again in the future.

Recap and Looking Forward

All in all I’m quite pleased with this little experiment so far.  In only a few hours I managed to set up a syndication framework for preserving and indexing content that would otherwise have become harder and harder to locate, and indeed start to establish a more efficient means of filtering and organising the data.

The fact the framework runs regularly checks for new content means the system is largely automated, and appears to therefore require a relatively low degree of maintenance.

That said there are issues to contend with, in the form of legal questions of copyright versus reuse, comment fracturing and preservation of existing conversational threads – and it may turn out there are additional issues (or benefits) that I have not yet considered.  It is my hope these topics will become clearer the more I contemplate and experiment.

Realistically this post could have been easily summarised in 140 characters on Twitter, and the discussion spread out across a collection of updates, however in light of the fact I want to document the responses for reference later I thought I’d devote a post to it instead.

One of the issues I’m finding with Twitter – even when using hashtags – is the fact many services only track Tweets for a certain period of time before they no longer show up in search results.  Therefore in order to retain access to hashtag discussions for a longer duration – ideally in perpetuity – it would seem that a different means of aggregating and syndicating Twitter updates is necessary.

For example, with #CCK09 and #ECI831 currently underway, there is already a plethora of activity and discussion emerging – much of it containing some valuable sources of information, references to resources, and even revealing trends that may not be immediately apparent or visible until later.  It would be really handy to retain a record of this – and perhaps even start to organise, interpret and repurpose the data.

What I’m hoping to hear from people are recommendations on existing sites or tools that provide this sort of service. Is there such a thing?

Otherwise it might turn out that we need to start looking towards establishing a syndication framework such as the combination WordPress/Feed WordPress plugin, which pulls in content from disparate sources and republishes it in another space.  That way we’d be able to retain Twitter discussions long after they’ve faded into the ether elsewhere.

Problem is, WordPress may not necessarily be the most ideal solution for this scenario because it’s meant for more long-form content.

On the otherhand, WordPress themes like P2 have emerged to adapt the WP framework to suit a slightly different purpose:

“A group blog theme for short update messages, inspired by Twitter. Featuring: Hassle-free posting from the front page. Perfect for group blogging, or as a liveblog theme. Dynamic page updates.”

Realistically one of the themes that has chararcterised the lifecycle of Twitter thus far is its organically evolving nature – largely in the wake of innovated usage trends amongst the user base.  From my recollection at least, use of the @ symbol to signify a reply to someone was inspired by the habits of users; likewise hashtags were as well.

In the case of tracking past updates, perhaps this is another element that Twitter was not originally conceived to do – with its primary focus being maintained on what is happening right now, and the trends that are emerging as we speak, rather than one has been said some time ago.  Yet this organically evolving nature is also an extraordinary opportunity to analyse and review the trends that emerge over time, and we’re only going to be able to do that if we continue to have ready access to the updates.

So after saying all this, I’m encouraging everyone to throw their ideas into the pot. 

What do you use to retain Twitter updates?  What could we use to preserve hashtag discussions?

Via @timbuckteeth I’ve recently run across a new tool for Twitter seems to have a lot of potential. TweepML is a service that lets you create indices of Twitter users that can made available to others.

The idea of cataloging groups of Twitter users is nothing new certainly, however the additional functionality that TweepML offers makes it a far preferable option to a standard web page or wiki article. In particular, whereas static indexes just provide links to view the profiles of other people on Twitter, TweepML actually lets you follow them as well – en masse if desired.

Creating Lists

The service itself is fairly simple. Just go to http://tweepml.org and create an account (it doesn’t appear to support open authentication through Twitter at present). Then click on the big orange button that reads “Get started with our TweepGen Tool – Add it to Your Site” to visit the table of your existing lists and click on the Create a new list link to access the list creation form.

[Note: Strangely it appears that the menu options (Home, My Lists, View Lists, Create a List) are not visible on the main page, only the TweepGen Tool area]

Users are added to this form simply by including each account name on its line without the @ symbol or URL link (e.g. mbogle).

Once you’ve created the list, you’ll have the option of choosing from a variety of buttons or links (see the Get Link area for your list), generating the XML for the list, or viewing the end product.

Sharing and Using Lists

Once a list is created you can circulate the URL to others. They will now be able to browse through the user list, follow some or all of them without having to go to Twitter to do it, as well as suggest new users, or request the removal of existing ones.

As an example of how this service might be useful, please see Alec Couros’ indices of students enrolled in EC&I Fall 2009 – Social Media & Open Education. As he explains:

“I have taken all of the Twitter user ids of for-credit students, non-credit students, and presenters and have created two http://tweepml.org/ lists. I had to create two lists as the limit is currently 100 members for each list. I am told by TweepML that they will raise the limit soon at which time I will combine the lists into one.”

A few minor criticisms…

Having said all this, I have a few minor criticisms and change requests.

First, TweepML does not currently support batch uploading of users – you have to enter them all by hand.  So if you intend to create a list that contains dozens or more people, there’s not quick way to do it.

Suggestion: Bring in support for CSV uploads, or incorporate other ways of making batch changes to lists.

Secondly, there also doesn’t appear to be a way to grant read/write access to other users and thus distribute the work of maintaining the list.  A single individual owns and controls each list, and is responsible for maintaining it.

Suggestion: Expand the management of lists to include a single owner, but additional nominated moderators/agents who can also make changes to them.

Finally, regarding notifications.  Users have the option to request additions or removals of users from the list, and this information is sent through as a notification to the list owner.  However near as I can tell, the information is only available in the email – there does not seem to be a record of the request maintained in the TweepGen Tool area.  So if you lose or delete the email, there is no way to track down the information it contained; and when you do know the changes, there is no quick way to approve them – it requires manual intervention.

Suggestion: Track requests in the TweepGen Tool area, and offer quick ways to either delete or process/activate the requests, rather than input them manually.

That said, these criticisms are minor ones and not intended to detract from the notion that TweepML is a useful service worth taking a look at.

To take a look at TweepML for yourself, just visit: http://tweepml.org

One thing I’ve just started experimenting with, which I’ve so far had very good results with, is the use of groups.  This isn’t a feature that exists in the native Twitter site unfortunately, but has been introduced by a couple of different Twitter clients – most notably TweetDeck and Seesmic.

Groups enable you to categorise collections of your Twitter contacts and effectively filter out everyone else.  Additionally, given both TweetDeck and Seesmic feature multi-column interfaces, you are able to configure one or more categories to be displayed along side the columns that contain the updates of all your contacts, mentions/replies and direct messages.

Effectively you can continue to view the ongoing stream of discussion that contains everyone’s updates, while ensuring that the people you really want to hear from aren’t lost in the shuffle.