In Depth: Data privacy: how safe is your data in the cloud?

We are using the cloud more and more. As Internet connections get faster and more reliable, the convenience of having all our data available on all our devices becomes ever more attractive.

However, there are disadvantages to using cloud services, particularly the free of charge ones that still have to make a profit somehow. There are many valid, albeit scary, questions you'll want to mull over before trusting a third party to keep your data safe and we've listed them below.

The answers, as you will discover, in this feature are generally not what you want to hear:

• Privacy - is your data stored or is it being mined for advertising and marketing purposes?
• Reliability - can you be certain that the service you're using will always be available? What guarantees do you have regarding the safety of your data and is there anything you can do to improve this?
• Security - is your data encrypted? Who has access to the encryption keys? Could your data be hacked or stolen?
• Continuity - Can the cloud provider suspend or cancel your account, possibly even losing all your data, for any reason?
• Performance - Is your Internet connection fast enough to use the services you want without delays?
• Copyright - who owns the content you upload? Can your photos be sold or published without your consent?

While a large company's security and backup policies are likely to be better designed and implemented than your own, you still have the questions about what they can do with your data. You may feel that allowing them to parse emails to provide more relevant advertising, which you may or may not ignore, is a fair price for a free service.

• Best cloud storage: Dropbox vs SkyDrive vs Google Drive vs iCloud

On the other hand, you may want to keep your private emails just that, and if you are using email for business, you've even more concerns. You are not only responsible for your own data but that of organisations you deal with.

You may also be concerned about keeping private information within your private network, not only for commercial secrecy but also data protection and preserving the privacy of those an organisation deals with.

One law for us, another for them

Another serious issue is that the cloud service may be operated in a different country to your own, and therefore subject to different laws and policies. Even if you are happy with the location, there is nothing to stop the cloud service provider moving everything to a data centre in a different country for economic or political reasons - or simply as a tax avoidance measure.

• Best cloud backup: how to choose the best online backup service

Google has come in for a lot of stick over its approach to privacy (not to mention its apparent consideration that paying tax is optional here in the UK) but it's are not alone in this, or even the worst. Google is simply the provider with the highest profile.

To its credit, it is reasonably open about using your data to make money - most of us knew Google would read our emails before we signed up for a Gmail account. Still, the idea of cloud services is appealing and convenient; so how do we resolve this dichotomy?

The answer is simple to run your own cloud. That way you get the convenience of easy access to your data from multiple platforms and locations but you retain control over that data.

Now, we are not suggesting that you set up your own data centre to implement your own Gmail, Dropbox and Facebook, but you don't need the scale of those organisations for a home or small business operation.

What do you need?

The cloud is basically built on top of the web, and Linux has something of a track record in providing web services. All you need is a Linux computer running the standard LAMP (Linux, Apache, MySQL, PHP) stack and you can install most of the software from the following pages in minutes.

That just leaves two questions to answer: what and where. Depending on how much you want to do, the hardware could be a small server box, your desktop computer if you leave it switched on all the time it may be needed, or even something really small and power efficient, like a Raspberry Pi or one of the plug computers. Any of these would be hooked up to your local network and so would give excellent speeds when used from within your LAN.

They could also be made available from the outside, but then you run into the question of the upload speed of your Internet connection. Most domestic Internet connections are asymmetric, giving far higher speeds from downloads than uploads. But accessing data from the outside world counts as an upload, so it would be slower - not unusably so, but it's something to be aware of, particularly if you want to support multiple users.

If you are providing access from the world at large, life is a lot easier if your internet provider is able to give you a static IP address, otherwise you will have to a dynamic DNS service. An alternative is a VPS, a Virtual Private Server.

This is basically a Linux distribution running on a virtual machine at a data centre, so you get the benefit of the fast speeds such environments provide, but it is still your own system and you control what software you include, and any passwords and encryption keys used, and the provider need not be able to read any of your data.You could also use a web host for this, as everything runs on a LAMP stack, but that reduces the options for protecting your data.

Whatever the hardware you choose, real or virtual, you need a full LAMP installation. If you are devoting a box to the job, it is simplest to install on the server-oriented distros, like Ubuntu Server, or something like Debian and select the web server option when installing it.

If you are using an existing system, installing Apache, MySQL and PHP should be sufficient. If you install the software we discuss from your distro's package manager, it will take care of this for you.

ownCloud

Keep your data secure - setup and run your own cloud server

Enough of the reasons why you may want to set up your own cloud service (or not), how do you go about it?

'The cloud' is a pretty vague term, covering just about any service that allows you to store your data 'out there' accessible via the internet. There are email services like Gmail, file storage services like Dropbox, photo galleries like Flickr, and the list goes on. You may not want to move all of these to a private cloud, the social sharing side of sites like Flickr makes the public aspect of them a benefit - especially as it offers a free 1TB of space now - but that's not always the case.

So where do we start, which service should we privatise first? One project that covers more than one of these areas is ownCloud. While it's main focus is for data storage, along the lines of Dropbox, it also provides a number of other features, some built in and some via plugins. If you want to play around with ownCloud before committing to installing it, try the demo at http://demo.owncloud.org.

This is a web application, so first you need to make sure your web server is working. If your computer has a desktop, you only need to check that http://localhost works in a browser. If you are installing on a separate server, use its hostname instead of localhost.

The latest version of ownCloud is 5.0.6. There are three ways to install ownCloud. If it is in your distro's repositories, install it from your package manager. Alternatively, download the latest tarball from http://owncloud.org/install/ and unpack it into your web root. This is the directory set as DocumentRoot in your Apache configuration. It's usually something like /var/www/ or /var/www/htdocs, although some distros use /srv/www instead of /var/www.

The third method is the web install, intended for remote installations, such as on a VPS. Download the web installer into your web space, then use your browser to open the PHP file. However you install it, you should now have an ownCloud directory in your web root, so point your browser athttp://hostname/owncloud and it will load the initial configuration page.

The main decision to make here is which database engine to use. SQLite is the simplest, and fine for single user installations. However, if you have anything else using MySQL, some of the other packages we will discuss later need it, then it makes sense to only have one database running and use MySQL for everything. Either way, you need to give the database name and user details.

An SQLite database will be created automatically, but MySQL users will have to create it before setting up ownCloud, which is covered in the MySQL section later on.

Keeping in sync

Once you've input the details of your administrative user and database, ownCloud will configure itself and you are ready to use it. However, there is one tweak you may want to make before getting going. One of Dropbox's limitations is the 300MB file size limit, ownCloud is limited to 512MB by default - but it is your cloud so you can change it.

The 512MB limit comes from the default PHP settings for maximum file upload sizes, you may want to increase this if using it on a fast network. Click on the username at the top right of the browser display to open the main menu and select Settings. You should be able to change the upload size setting to whatever you need, but there is a bug in the current release, probably down to PHP, that causes problems when you set this to 2GB or higher, reporting that the file is too large when it is not, so set it to a maximum of 1.9GB until this is resolved.

You now have a Dropbox-like file storage area, but what about syncing it with your devices? To do this, you need the ownCloud client installed. This is available for all popular OSes, plus Windows and Mac OS X. It's usually in the distro package managers as something like owncloud-client, but some distros still list it as mirall, the previous name for the sync software.

Installing and running it puts an icon in your Task Bar. The first time you run it the client asks for the address of the server and the directory to sync with it. So far, it works just like Dropbox, but there is more. You can add extra sync jobs, either using different directories or even different servers.

You are not limited to syncing a single directory with a single server. This not only gives the flexibility of being able to sync data from different folders, but it also means you can sync, for example, work files and personal files with different servers - complete privacy and more features, a double win!

There are also sync clients for Android and iOS, which are installed in the usual way.

File sharing

What about sharing files with others? When you hover the mouse over a file in the web browser, a menu appears to the right which includes a Share option. This has a box for giving an email address with which to share the file, but if you tick the Share with link box, it gives you a URL that you can email or otherwise pass on to others. This is a download link, which doesn't expose the contents of your cloud to others but lets them download that file or view (and download from) a directory if that's what you shared.

If you have multiple users and you don't want them sharing files they shouldn't, you can add users to groups and select the option in the admin page to only allow them to share with users in the same group. This allows them to collaborate without giving away trade secrets.

Users are created from the Users option on the main drop-down menu. As well as assigning them to groups, you can also limit the amount of storage space they may use.

There is another way of accessing your files, using the WebDAV protocol, which is supported by file managers and other programs. This enables you to access files in your cloud as if they were local. The exact syntax used varies; in Nautilus you would use dav://your.server/owncloud/files/webdav.phpwhile KDE prefers webdav:// to dav:// and LibreOffice likes plain old http://. That's right, LibreOffice will load files direct from your cloud.

Because WebDAV is a two-way protocol, it also allows saving of files, which neatly removes many of the reasons for a separate web-based office client - you can use standard tools with files in your cloud. Some of the ownCloud modules, such as the Calendar, Contacts and Bookmarks apps, start off empty even though you probably have all this information stored elsewhere.

Fortunately, it is easy to import from standard formats like ICS, VCF and HTML respectively. It is also possible to set your Calender to sync with other devices, press the settings button to see the URL for syncing with other calendar clients. The music and Photos apps will find suitable files in any folder of your cloud. You don't need to have them neatly organised, although that's no reason not to.

In the Music app you need to press the Rescan button after uploading new file to have them added to the list.

Modules and plugins

When you get tired of exploring the available features of ownCloud, you can add some more. There are a number of plugins or apps in ownCloud terminology, that you can enable to add features.

Some of these are installed by default while others can be downloaded from http://apps.owncloud.com. Select Apps from the drop-down menu to see the installed plugins. The ones in bold at the top of the list are enabled. To use one of the others, select it and click the Enable button.

The More apps button takes you to apps page where you can browse or search the apps. Downloading one gives you a ZIP file that you unzip into the apps directory of your installation.

There are some extra features in ownCloud that make it even more appealing. For example, file versioning. If you overwrite a file with a newer version, the older file is kept and is still accessible, providing you have enough space available. There's also a Trash bin for deletion, so any files deleted using the web interface can be retrieved if you change your mind. This is a relatively new program, so releases are coming thick and fast.

There's also more chance of a security vulnerability being discovered in less mature code, so it is safest to always keep it up to date. The Admin page warns you if there's a newer release than you have installed.

After upgrading, you may find that ownCloud gets stuck in maintenance mode. This is to prevent anyone accessing it until you, as administrator, have performed any upgrade steps. Editconfig/config.php and change the maintenance setting from true to false. Reload the main page, answer any questions it may ask and your cloud will be running again.

Webmail

Keep your emails private - setup your own webmail service.

There's a lot you can do with ownCloud, but there is one important cloud service it does not touch and that's email. Of all the cloud services, the one that generates the most concerns over privacy is webmail in general and Gmail in particular. Your emails are transmitted and stored as plain text and Google openly admit that it reads your email in order to provide you with targeted advertising.

For non-confidential mail, this is acceptable to most people, but you can run your own webmail service if it doesn't. It gets better, you don't even need to change your email address or have a static Internet address as running your own SMTP server generally requires. Webmail programs do not have to run on the same system as the mail server, so you can install webmail on your own server and continue to use your current email address and server.

There are several webmail alternatives, two of the most popular are SquirrelMail and Roundcube, both are written in PHP and run on the usual LAMP stack. We will look at Roundcube here, but Squirrel mail is just as easy to set up and just as capable. It is quite feasible to run both.

They are only email clients after all and plenty of people have more than one email client installed. Both webmail services use IMAP to communicate with the mail server, which means the emails stay on the server, allowing you to also read the same email with a desktop or mobile email client as well.

IMAP has another advantage over the older POP3 protocol, beyond removing the need to download everything before reading, it stores information such as which emails you have read on the server, making it easy to switch between mail clients and still keep track of where you are up to.

Installing webmail

Installing Roundcube is simple. You can either do it through your package manager or directly onto your server by downloading the tarball from www.roundcube.net. Unpack the tarball into the root of your web server. This creates a directory called roundcubemail-0.9.0 (for the current version). Either rename or symlink this to something more usable, like roundcube or even webmail.

Roundcube needs to write to files in its logs and temp directories, so make sure these are owned by the user running the web server, usually apache or www-data. Then run the installer by pointing your browser at http://address-of-server/roundcube/installer.

Make sure that none of the checks show Not OK. Missing optional modules are fine, and you only need one database available (Roundcube supports several database formats).

One important setting it checks is PHP's date.timezone option, which needs to be set in order for your mails to have the correct timestamp. If you are using a VPS rather than a local server, this should be set to the timezone for wherever the server is physically located. The setting is found in/etc/php5/apache2/php.ini and the line should be uncommented by removing the leading ; and then the timezone added.

You will usually find the correct timezone specification in /etc/timezone. You need to restart Apache after changing this, or any other PHP settings, with: sudo apachectrl restart

Setting things up

Reload the installer and correct any further errors before proceeding. The next page sets up the configuration. If you are using your own webmail because of privacy concerns, you will probably want to disable the spell checker, because the default one sends the text to Google for checking!

You need to tell it about your database and, unless you are using SQLite, you need to set up the tables yourself. You can do this by running the mysql client as root and issuing these commands:

CREATE DATABASE roundcubemail; 
GRANT ALL PRIVILEGES ON roundcubemail.* TO username@localhost IDENTIFIED BY 'password'; FLUSH PRIVILEGES;

Change the username and password to suit, but leave the localhost because the database server is on the same system as the web server. Alternatively, you can create the database and user with phpMyAdmin.

Now you'll need add one or more IMAP servers to connect to. If you add more than one, you get a menu to choose from when you load the page, if you leave this blank, you have to type in the name of the server each time you want to use Roundcube.

When you click on the Create Config button, you see a page with links to download two files, main.inc.php and db.inc.php. Copy these to the config directory of your Roundcube installation. Then click on Continue and your configuration will be tested. Correct anything that is flagged up, either by going back to the previous page or by editing the files directly, and reload the test page.

You should also test the SMTP and IMAP settings here. Note that the big red warning at the end of the test page, make sure you remove or disable the installer once Roundcube is set up, otherwise anyone can see your settings and passwords.

Reading your mail

The page that you'll see when you load your webmail URL into your web browser depends on your settings. If you've set up a single mail server, it will go straight to asking for your username and password, otherwise you will see either a menu to select between the different servers you have set up or a text box if you have added no servers.

The first time that you login there will be a delay as your mailbox is scanned. The speed at which your mailbox loads depends mainly on the speed of the connection you have between the web server and your mail server, running them on the same computer makes a big difference, and the amount of junk, sorry mail, you hang on to.

The settings menu contains options to change both the behaviour and appearance of Roundcube, while the website at http://roundcube.net has a wiki with plenty of information on tweaking Roundcube to suit your needs. There are also plenty of plugins available from this website, which can be installed to enhance the features and appearance of Roundcube. Plugins are supplied as archives, usually TAR or ZIP, which should be unpacked into Roundcube's plugins directory - you will find there already several present with a default installation.

Plugins are disabled by default, you enable them by adding their names, as they appear in the plugins directory, to the $rcmail_config['plugins'] array in config/main.inc.php. For example, to add the included autologon plugin and the 3rd party SpamAssassin plugin, the setting would look like:

$rcmail_config['plugins'] = array('autologon', 'sauserprefs');

Social Networking

Ally your privacy concerns - establish your own social media platform.

Setting up webmail is all well and good, but email is all a bit 20th century, at least that's what the likes of Facebook would have us believe! Social networks raise several privacy concerns and while companies may use the likes of Facebook to 'engage' with consumers who 'like' them, they wouldn't use it for internal communications. In fact, most organisations have strict policies on discussing any company business on social networks.

However, these services do provide a valuable way for people to interact, either in public or within an organisation. What we need is a social networking platform that we can run ourselves, preferably one that is open source, to allay privacy concerns.

There are a few options, such as Buddy Press which started life as a WordPress mod and grew into a complete social media platform, but we are going to look at Elgg.

Looking at the list of well-known organisations and educational establishments that choose this particular platform gives you an idea of its popularity and suitability.

A familiar procedure

The installation process is similar to many other web applications. Download the ZIP file, unpack it into your web server's document directory, preferably making a symlink to a more useful name than elgg-1.8.15, and set up the database.

Elgg uses MySQL so the standard instructions given elsewhere apply for creating a database and user. You also need to create a data directory for Elgg, this should not be in your web server's DocumentRoot, the area from which it serves pages, as you do not want the contents to be directly accessible from a browser.The directory should be owned and writable by the user running the web server, usually www-data or apache.

That's the command line bit done, although you could do most of that with phpMyAdmin and an SSH-aware file manager, now load http://yourserver/elgg into your web browser. This guides you through the set up, making sure you have all the modules you may need, setting the database credentials and testing access and configuring the location of the data directory.

If it picks up a problem at any point, it tries to give a helpful hint as to the solution. Hit refresh after fixing the problem to run the checks again.

The final step is to create an administrator account. When you log in with your administrator account, you can greatly change the appearance of the site as well as enabling various plugins and other options.

Making Elgg your own

Elgg is well documented, but those documents are well hidden, with no link from the home page of the Elgg website. Prior to installation, go to http://docs.elgg.org. Once Elgg is installed there are Manual and FAQ links at the bottom of the Administration page. The method of controlling plugins should be familiar by now.

Installed plugins are accessed through a link on the administration page. New plugins are downloaded as ZIP files to be unpacked into the mod directory, after which they appear in the aforesaid list.

Themes are also plugins, to be installed and configured in the same way. Links to the theme and plugin repositories are on the front page of the administration manual. Bear in mind that as themes are treated like plugins, it is possible to select more than one at once, with sometimes strange results.

A useful feature of the plugins page is that you can move them up and down the priority-sorted list, changing the order in which they are loaded.

What is LAMP?

Web services run on what is known as a LAMP system, where the first three letters stand for Linux, Apache and MySQL while the P could be PHP, Perl or Python, depending on your preferences.

Apache is the de facto standard web server for Linux, although there are alternatives, such as Lighttpd and Cherokee, that are often lighter and work with these services. However, Apache is the most popular and everything comes set up to work with it, whereas using a different web server may involve some more work, details of which can often be found on the projects' websites.

Installing Apache is easiest from your package manager (you have already taken care of the L in LAMP, right?), Python and Perl are often installed by default so that leaves PHP and MySQL, both installed through the package manager too.

There's more

We have looked at three main programs here, covering file storage, email and social networking, although they all provide extra features, such as photo albums, calendaring, contacts and bookmarks management. There are alternatives to each of these, as we have mentioned, but there are plenty of other cloud type programs around.

For example, Gallery is a dedicated photo album that provides far more features than the addons for the programs we have looked at here. It is a typical web application using PHP, so the installation and configuration procedure is much the same as the other programs, and it has a comprehensive web administration interface for creating public and private albums, as well as many available plugins to enhance Gallery even further.

There is also an open source instant messaging service that uses Jabber (which is used by Google Talk among others).

Asterisk, which we have covered before, can be used to set up your own VoIP gateway, which is especially useful for an organisation with more than one office.

There is only one important area where the open source alternatives do not currently match up to the commercial offerings and that's office productivity software. While the WebDAV accessibility of ownCloud does allow editing anywhere, and by anyone with the correct login, it still depends on client-side office software.

There is a project in development called OX documents that currently has a demo at https://www.ox.io/ox_text. Interestingly, it supports ODF as well as Word format files and is the first step towards an open source online office suite.

The first release is due around now and when it adds a spreadsheet, we will be able to do just about anything in our own clouds that we can in the proprietary ones, except lose our privacy.

Adding a MySQL DB

MySQL is a database server, all of the packages we have mentioned here need some sort of database to hold settings and data. While MySQL is one of the most popular, there are alternatives.

SQLite is, as the name suggests, a much lighter option. It stores a complete database in a single file and works well on single user systems that do not have particularly demanding requirements. However, it does not scale particularly well, either in terms of users or load.

Because MySQL uses the classic server/ client model, one server will take care of the database needs of several programs, so if you are using more than one of the packages covered here, MySQL is probably the best choice. The disadvantage is that you need to set up databases and users on the server for each package.

When you install MySQL it asks for a password, this is for the root user and not to be used by individual packages. You can create a new database and a user for it using the command line MySQL client mysql -uroot -p will prompt for a password, then you create the database and user with:

CREATE DATABASE dbname; 
GRANT ALL PRIVILEGES ON dbname.* TO username@localhost IDENTIFIED BY 'password'; 
FLUSH PRIVILEGES;

The first line creates the database, the second creates a user with a password, allowing them full access to that one database. The flush command simply tells MySQL to implement the changes immediately.

When you've finished \q will exit the client. You can tell your program to use that database and user and it can create the database structure itself.

An alternative to mucking around with all this command line malarkey is to use phpMyAdmin, a web-based administration tool for MySQL. There's a certain irony that you actually need to use the command line to create the user for phpMyAdmin, but what you need to do is all detailed in the web interface.

Openstack

While we have concentrated on ownCloud here, there are alternatives; one such is OpenStack. However, OpenStack is aimed more towards large scale use. OwnCloud can be used in enterprise deployments and there are commercial variants available, but it is also well suited to personal and small organisation use.

On the other hand OpenStack is aimed more at those wishing to offer cloud services commercially. That certainly doesn't rule out OpenStack and if your requirements are for something more than a personal cloud service, you should take a look at www.openstack.org to see if it better suits your needs.