Pilots and Enterprise Services

One of the topics that’s really been at the forefront of what I do these days has been the notion of pilot programs, and an emerging discussion of what they look like, how you plan and implement them, how you evaluate and assess their outcomes, and what their scope and scale is or perhaps what it should be.

I have fairly established opinions on this whole topic, which have gradually evolved over time; but I don’t think I’ve ever taken the time to actually articulate what they are. So for my own benefit I thought I might do that now.

That being said, this is only my first go, so I will most likely elaborate on things more over time.

Background

As a bit of a background here, I work in a central unit in the university, and one of the main missions of the team I’m on is to the Research, Evaluation and Development of technologies that can enable and enhance learning and teaching. As such we’re the R.E.D. Team in the TELT group, in Learning & Teaching @ UNSW (make sense?).

Now there is a lot of technology out there; a ton of tools, software and programs that do different things and MIGHT be useful for learning and teaching purposes. So it’s our job to identify the ones in particular amongst the diversity of the landscape that will be explored for potential release to the wider academic community, and then having identified potential candidates, work with central IT to establish pilot systems and conduct evaluations; and if all goes well, to transition them to enterprise level services.

What I mean by “centrally supported”

Now here I should point out, when I refer to the notion of something being “centrally supported” I don’t mean there is one ring to rule them all, and we hand our decisions down from on-high at which point they must be immediately implemented by the masses – and exploration of local alternatives is not allowed. I see technology in learning and teaching as being highly dependent on context – so I’m not the least bit interested in telling people how a tool must be implemented.

No, when I say “centrally supported” I’m referring to the more pragmatic, practical aspects – not the pedagogical ones. For example, the supporting infrastructure for centrally supported services are hosted and maintained by central IT – this includes the servers, updates, and tech refresh of the hardware. It also involves user and technical support when things go wrong, bugs are identified, or people are generally unsure of how to use the technology.

My unit does play a role in the professional development process, and facilitates discussions on how to actually use the technology to enhance the learning and teaching processess – but this is far more of a conversation between colleagues that aims to foster a culture of sharing in which individual instructors explore what the technology means to them and in turn shares what their experiences have been.

Pilots

The notion of pilots then is a very specific process that seeks to ensure that what is being released is

1. Of potential or recognised value to the academic community;
2. Technically reliable and scalable (e.g. works as you expect it to and can handle a load of potentially thousands of people);
3. Integrates fairly well with existing systems (HR, Student Admin, and others); and
4. Can be supported centrally.

DIY edtec

Again, this process seeks to identify what can be implemented at an enterprise level, not to dismiss or cast judgment on projects that are more strategic to localised groups. As far as I’m concerned, if you find a service that works brilliantly for your local purposes but isn’t necessarily useful to the wider institutions – go for it. The only consideration is that you’ll need to run it off of your own back – central IT can only do so much.

This is after all what I do with all of my web projects. This blog, YouTube, Twitter, SlideShare, Skype, etcetera – I look after them all myself and am responsible for sorting out my own issues. They work brilliantly well for my purposes and I’m keen to use them; but I don’t expect central IT to resolve problems or issues I run into when using them.

In many regards this is the essence of what DIY edtec is all about. In personal uptake lay a tremendous amount of flexibility and freedom to explore and use whatever you like, regardless of what other people think. You dictate your own destiny; but you also look after your own issues.

How this relates to pilots

Incidentally, I say all this to try and clarify what may appear to be a conflict between what I say about things like edupunk, and what I’m doing in the name of central services. Not everyone is interested in capable of running their own show from the standpoint of educational technology; that’s where centrally supported services come in. The issue with centrally supported services, though, is they have to cater to the needs of literally thousands of people. So by necessity the way they’re organised, implemented, and maintained is far more bureaucratic and convoluted.

This is particularly visible when it comes to pilots, since by necessity we have to be more detailed about planning, implementation, documentation and evaluation. In my use of DIY edtec, I am constantly evaluating the value of a platform as I go, based criteria that I alone dictate; but I’m just one person. When your userbase is potentially 30,000 people, what you pay attention to and the criteria you use to determine success or failure is monumentally more complicated.

Categories of Activity

So when it comes to pilots I personally look at several different categories of activity:

• Pedagogy – does the technology suit the identified needs of the learners and teachers; does it also offer freedom and flexibility to both instructors and students; can it support diversity in learning habits
• Technology – is the platform reliable; is there vendor support; in the case of open source, is there a reliable release cycle for patches and updates as well as robust community support; is there potential to integrate with existing institutional systems; do APIs exist to ease or automate administrative tasks.
• Maintenance - does the underlying infrastructure align with existing systems and expertise; how labor intensive is the maintenance and management
• Support - what are the support implications; is the framework very complex, which may lead to increased demands for assistance; does it have complex dependencies and requirements for end-user systems

Why pilots are important

While certain elements will be clearly apparent without introducing a technology to a user sample, there is a great deal that cannot be ascertained without introducing real-live people into the mix. This is why pilots are so important. They enable us to examine the true implications of a technology, and yet do so on the small scale under very controlled circumstances with clear boundaries for start and finish.

If the evaluation of the pilot comes back positive, then you can begin to introduce the technology more widely. However if the pilot is a disaster, the impact will be quite low relative to the problems with an untested enterprise level application. When things go wrong in a sample of a few hundred people, it’s difficult but manageable. When they go wrong in a population of 30,000 people, it’s disastrous.

Pilots are crucial because they expose potential frameworks to a systematic layer of research, evaluation and development, and thus ensure they are valuable, appropriate, and sustainable for use on the large scale.

More to contemplate…

This is just my first pass at articulating my thoughts on pilots; I expect to elaborate more over time.  In the meantime if I’ve overlooked anything glaring or you have your own experiences, research or resources to share PLEASE DO! I’m all ears.