Name the File Extension

The Days of Tundra are drawing closer and we're facing some interesting decisions. One of them is the file extension for Tundra scenes, now that sharing full scenes (and parts of them) is so simple. Toni Alatalo wrote the text below on the realXtend mailing list and if you have opinions, please contribute. Anyway, here's Toni:

I think we are really close to being able to release a kind of preview demo of Tundra, i.e. Naali with the server module & executable. One key is that Jukka wrote a nice doc about how to use the document/scene/application files and explanation of what they are, and I changed the public doxygen to use the version from Tundra branch so the page is up at http://www.realxtend.org/doxygen/tundradocumentfiles.html . I think we basically just need to write a little more usage docs to at least point to where the example scenes are and make an installer.

As you can read in the doc, the local server works nicely as a preview / editor thing -- it is not only for people who want to host servers, but also for e.g. modellers, texture artists and scripters to easily see how their things look and work in ReX. The server executable is a normal Naali app, shows the scene using Ogre etc (but you can optionally run it without gfx for server usage). A bit like the local scene preview in Naali now, but much nicer and faster 'cause you don't need a server connection anywhere -- just run Naali standalone to view local files. By clicking a file in your file manager so it starts Naali showing that scene. Besides these own document files in the internal format, you can of course also import dotscene files as well, and there's support for not only Ogre meshes but Collada too etc.

There is one non-technical issue remaining, and I feel a bit stupid to bring it up 'cause is kind of nitpicking, but it is something we should get right the first time so is worth some consideration now. It is the file name extensions that I was asking about in sprint planning as well. We talked about with Antti yesterday but didn't conclude and he suggested posting here to get ideas & feedback, so here we go. Because the issue is non-technical and I'd like to hear user opinions, decided on last minute to post this to users list instead of the -dev list.

Currently, like that doc says, we use 'txml' and 'tbin' for so-called Tundra files. Previously they were just .xml and .bin but the guys added the t* to make them unique for registering to operating system so that opening them directly to the right application works. There is a couple of problems with these names:

1. That entity-component serialization system is not really Tundra specific, is not in the server module and not tied to any protocol. It is implemented in Naali core and was originally and will used with Taiga (to store Naali EC data on opensim, started in last March or so). I've been testing the idea of calling the format the 'realxtend format' instead, and it seems to make sense. Matti K. at least agreed in the meeting. The counterlogic here goes that Tundra is the name for the design, the legacy-free usage of pure EC data for making everything without things hardcoded in e.g. LLUDP / SL assumptions. And that Tundra is a strong nice sounding name! With this logic if there are some day other implementations that support the Tundra way, they also implement the Tundra protocol and the support for Tundra files etc. .. e.g. a modtundra to opensim? This might be confusing though 'cause otherwise Tundra is the name for the server module implementation in Naali. One funny point with the current 'txml' and 'tbin' names is that 'cause Taiga also start withs T, we could say they are both Tundra and Taiga files :)

2. Erno argued that there are also many other XML (and of course binary) files used with Tundra (i.e. Naali), and I think that's a good point. For example the module loading configuration files are xml, in modules/core/*.xml -- those could be called 'tundra xml files' as well. It would be good to say what is in the file in the name, and in one way it is the scene. Jukka's doc also says "a scene file". So Erno was thinking .rts for RealXtend Tundra Scene could be it, which is logical enough but I don't think that sounds too great :o (even though one idea with the generic EC model is to allow making Real-Time Strategy games :)

I was now thinking of these extensions again, but now with better logic:

.rex - RealXtend Entity XML (earlier just thought it's RealExtendXml :p)
.rxb - .rex binary (rxb just sounds like 'rex in a tight binary form', doesn't it?-)

The files are exactly the entities, the whole idea of the formats is to store the entities, either a full scene or just some selected entities. There is nothing else in the files, not for example assets like Collada .dae files can have (dae is 'digital assets exchange'), nor some generic Tundra config stuff .. only the entity-components with their attribute values.

Opinions? Please anyone tell yours, this is for end users, you don't need to be a dev to be a stakeholder .. these are the files you are gonna be using to do work with your stuff!

For folks familiar with OpenSim files, .txml/.rex files are like the files that save-xml2 writes -- have the full scene, with asset references, without assets themselves. To make a bundle with assets like OARs are, you can simply make a zip with e.g. a folder with the assets. I guess we must come up with a name for these zips later too, like OAR is (they are tar gzips). If OpenSim gets the generic EC stuff to core some day, then OAR and tzip/rexzip files may become the same.

The Tundra Project

When the realXtend project started in the fall of 2007, two different technologies were chosen as the starting point for development; the OpenSimulator server (based on reverse-engineered Second Life servers) and the open source Second Life viewer. The platforms were chosen as the best available candidates considering the rapid prototyping necessitated by the project’s six-month schedule. Even in hindsight the choices were good, despite the negative issues encountered further along the development.

The prototype produced during the first stage of the project proved successful technology wise and it gained a fair amount of positive publicity. The licensing and technology of the Second Life viewer had been somewhat problematic from the start and the situation was getting worse with increasingly ambitious technological goals for the project. It was also obvious that the licensing (GPL, or rather the interpretation of GPL in the open source virtual worlds community) would make it very difficult to form a strong community of developers around the project.

Early 2009 saw the emergence of the Naali project. Naali (Finnish for arctic fox) is an entirely new viewer built from scratch (of course utilizing existing suitably licensed material where possible) with the intention of eventually replacing the Second Life –derivative first generation realXtend viewer. During 2010 this goal was achieved, currently Naali has almost all relevant functionality of the earlier viewer and surpasses it in a number of critical categories.

To make server development easier, the realXtend server side enhancements were modified into a module called “ModreX” and a suitable module interface was created to OpenSim. The ModreX module provided more flexibility on the server side and the OpenSim project itself made progress towards its goal of being a flexible platform for virtual worlds hosting. Even so, it still remains a relatively heavy service in terms of hardware resources, requiring several ports open in firewalls, database installations etc. Despite an increased level of flexibility, adding support for the core Naali features (namely the entity-component based content authoring) still wasn’t simple by any means.

The Tundra Server

The development of the Naali viewer progressed at a good pace and in the summer of 2010 initial research indicated that it would be fairly straightforward to add a server module to the viewer. The Naali server module was named “Tundra” and the development work was started in early autumn of 2010. With this approach it was possible to create a perfect counterpart for Naali, since much of the Naali code would be used as is. Also much of the programming work done in the future would be usable on both the viewer and the server side. The approach is quite common in for example many contemporary games and offers some intriguing advantages.

One of the immediate benefits is the “single user mode” or the ability to view content directly with the Naali viewer, without a connection to a server. At the same time it becomes possible to share content as well, which in practice means the availability of “ad hoc servers”, much like in games that allow users to host multiplayer events. In comparison, OpenSim (or Taiga) is ideologically more like a massively multiplayer online game that runs on a persistent dedicated server. The former can be made lighter and simpler to deploy, which makes it suitable for a number of applications. A notable difference is the new protocol implemented in Tundra; after researching several possibilities the developers decided that the fastest way to move forwards with the project was to use a BSD-licensed protocol known as “Kristalli”.

Naturally it is also much simpler to experiment with new features when the codebase is shared by both the server and the viewer and the current developers are of course already familiar with the Naali code. The API is also shared by the server and the viewer, making it possible to utilize the same Javascript or Python code in both. All of these features are intended to make creation and deployment of diverse applications simpler and more cost-effective.

The Second Life architecture and thus to an extent OpenSim, have a fairly specific scene model based on separate primitive objects, avatars, terrain etc. realXtend has a flexible entity-component based architecture, whose purest implementation exists in the Naali / Tundra combination, which is directed more towards being a generic 3D application platform instead of a specific kind of a virtual world. For creating large multiuser virtual environments OpenSim offers many advantages, like for example safe sandboxed LSL scripting for user-to-user content.

Interoperability between different worlds is an important question. The obvious first step is the ability to create content in ways that make it possible to utilize it in different environments. Tundra supports a number of different 3D model formats, making it backwards compatible with Taiga (OGRE meshes), Tundra also has support for the Immersive Education Initiative’s chosen open library format (Collada). The next step for interoperability is a universal authentication / identity. Tundra is designed so that adding different authentication schemes (including none or just a simple ad hoc password authentication) should be simple. The most difficult challenge is functional interoperability, currently there is no simple answer. Tundra has support for popular Javascript and Python scripting engines, but true interoperability in scripting will require a lot of work and co-operation from all virtual worlds providers.

Perhaps most important of all, Tundra is a demonstrator for the new Naali-style architecture and content creation features. Building the features on top of OpenSim based Taiga would have been more work, but not impossible and as Tundra matures, it will act as a reference implementation anyone interested in its content creation abilities can learn from. It’s also important to note that despite all the differences, Tundra and Taiga still have similarities and many of the features are already supported on both platforms. Thus adding full support for the new features and full compatibility for both platforms is well within the realm of possibility, OpenSim internals permitting.

Blender, Python & realXtend

Pyppet has published a blogpost on his realXtend and Blender related work here. A pretty interesting read for the more technically oriented readers.