Cloud / Datastore Support¶

Cloud abstractions and helpers.

More generally, this is “datastore” support, since the basic interface could support any interface, file, memory, cloud, etc. But, as this is called a “cloud” browser, we’ll call it the “cloud” module.

Configuration¶

Cloud configuration.

class cloud_browser.cloud.config.Config¶

General class helper to construct connection objects.

classmethod from_settings()¶: Create configuration from Django settings or environment.

classmethod get_connection()¶

Return connection object.

Return type: cloud_browser.cloud.base.CloudConnection

classmethod get_connection_cls()¶

Return connection class.

Return type: type

Errors¶

Common cloud error wrappers.

exception cloud_browser.cloud.errors.CloudException¶: Base cloud exception.

class cloud_browser.cloud.errors.CloudExceptionWrapper¶

Exception translator.

This class wraps a “real” underlying cloud class exception and translates it to a “common” exception class from this module. The exception stack from the wrapped exception is preserved through some sys.exc_info() hackery.

It is implemented as a decorator such that you can do something like:

class MyWrapper(CloudExceptionWrapper):
    '''Convert exception to another one.'''
    translations = { Exception: NotImplementedError }

@MyWrapper()
def foo():
    raise Exception("Hi.")

foo()

The alternate way is to handle the translations directly in a member function:

class MyWrapper(CloudExceptionWrapper):
    '''Convert exception to another one.'''
    def translate(self, exc):
        if isinstance(exc, Exception):
            return NotImplementedError

        return None

@MyWrapper()
def foo():
    raise Exception("Hi.")

foo()

which produces output like:

Traceback (most recent call last):
  File "...", line ..., in <module>
    foo()
  File "...", line ..., in wrapped
    return operation(*args, **kwargs)
  File "...", line ..., in foo
    raise Exception("Hi.")
NotImplementedError: Hi.

So, we can see that we get a different exception with the proper stack.

Overriding classes should implement the translations class variable dictionary for translating an underlying library exception to a class in this module. See any of the data implementation modules for examples.

classmethod excepts()¶

Return tuple of underlying exception classes to trap and wrap.

Return type: tuple of type

classmethod lazy_translations()¶: Lazy translations definitions (for additional checks).

translate(exc)¶

Return translation of exception to new class.

Calling code should only raise exception if exception class is passed in, else None (which signifies no wrapping should be done).

exception cloud_browser.cloud.errors.InvalidNameException¶: Bad name.

exception cloud_browser.cloud.errors.NoContainerException¶: No container found.

exception cloud_browser.cloud.errors.NoObjectException¶: No storage object found.

exception cloud_browser.cloud.errors.NotPermittedException¶: Access is not permitted

Datastores¶

Cloud Browser is written with a pluggable backend datastore model in mind. Currently, there are datastore implementations for a basic filesystem and cloud (e.g., Rackspace, Amazon) backing stores. Other cloud stores shouldn’t be too hard to port over to this app, as Rackspace has probably the most little extra “gotcha’s” in listing objects using implied / pseudo- directories.

Abstract Base Class¶

Cloud datastore API base abstraction.

class cloud_browser.cloud.base.CloudConnection(account, secret_key)¶

Cloud connection wrapper.

cont_cls¶

Container child class.

alias of CloudContainer

get_container(path)¶: Return single container.

get_containers()¶: Return available containers.

max_list = None¶: Maximum number of containers that can be listed or None.

property native_conn¶: Native connection object.

class cloud_browser.cloud.base.CloudContainer(conn, name=None, count=None, size=None)¶

Cloud container wrapper.

get_object(path)¶: Get single object.

get_objects(path, marker=None, limit=20)¶: Get objects.

max_list = None¶: Maximum number of objects that can be listed or None.

property native_container¶: Native container object.

obj_cls¶

Storage object child class.

alias of CloudObject

class cloud_browser.cloud.base.CloudObject(container, name, **kwargs)¶

Cloud object wrapper.

property basename¶: Base name from rightmost separator.

property is_file¶: Is a file object?

property is_subdir¶: Is a subdirectory?

property native_obj¶: Native storage object.

property path¶: Full path (including container).

read()¶: Return contents of object.

property smart_content_encoding¶: Smart content encoding.

property smart_content_type¶: Smart content type.

type_cls¶: alias of CloudObjectTypes

class cloud_browser.cloud.base.CloudObjectTypes¶: Cloud object types helper.

Filesystem Datastore¶

File-system datastore.

class cloud_browser.cloud.fs.FilesystemConnection(root)¶

Filesystem connection wrapper.

cont_cls¶

Container child class.

alias of FilesystemContainer

class cloud_browser.cloud.fs.FilesystemContainer(conn, name=None, count=None, size=None)¶

Filesystem container wrapper.

property base_path¶: Base absolute path of container.

classmethod from_path(conn, path)¶: Create container from path.

get_object(path)¶: Get single object.

get_objects(path, marker=None, limit=20)¶: Get objects.

obj_cls¶

Storage object child class.

alias of FilesystemObject

class cloud_browser.cloud.fs.FilesystemContainerWrapper¶: Exception translator.

class cloud_browser.cloud.fs.FilesystemObject(container, name, **kwargs)¶

Filesystem object wrapper.

property base_path¶: Base absolute path of container.

classmethod from_path(container, path)¶: Create object from path.

class cloud_browser.cloud.fs.FilesystemObjectWrapper¶: Exception translator.

Apache Libcloud Datastore¶

ApacheLibcloud datastore.

class cloud_browser.cloud.apache_libcloud.ApacheLibcloudConnection(provider, account, secret_key, host=None, port=None, secure=True)¶

ApacheLibcloud connection wrapper.

cont_cls¶

Container child class.

alias of ApacheLibcloudContainer

wrap_libcloud_errors = <cloud_browser.cloud.apache_libcloud.ApacheLibcloudExceptionWrapper object>¶: Exception translations.

class cloud_browser.cloud.apache_libcloud.ApacheLibcloudContainer(conn, name=None, count=None, size=None)¶

ApacheLibcloud container wrapper.

classmethod from_libcloud(conn, container)¶: Create container from libcloud.storage.base.Container.

get_object(path)¶: Get single object.

get_objects(path, marker=None, limit=20)¶: Get objects.

obj_cls¶

Storage object child class.

alias of ApacheLibcloudObject

wrap_libcloud_errors = <cloud_browser.cloud.apache_libcloud.ApacheLibcloudExceptionWrapper object>¶: Exception translations.

class cloud_browser.cloud.apache_libcloud.ApacheLibcloudExceptionWrapper¶

ApacheLibcloud cloudfiles exception translator.

classmethod lazy_translations()¶: Lazy translations.

class cloud_browser.cloud.apache_libcloud.ApacheLibcloudObject(container, name, **kwargs)¶

ApacheLibcloud object wrapper.

classmethod from_libcloud(container, obj)¶: Create object from libcloud.storage.base.Object.

Boto-based Datastore Abstract Base Class¶

Abstract boto-based datastore.

The boto library provides interfaces to both Amazon S3 and Google Storage for Developers. This abstract base class gets most of the common work done.

Note

Installation: Use of this module requires the open source boto package.

class cloud_browser.cloud.boto_base.BotoBucketWrapper¶

Boto boto bucket exception translator.

error_cls¶: alias of cloud_browser.cloud.errors.NoContainerException

class cloud_browser.cloud.boto_base.BotoConnection(account, secret_key)¶

Boto connection wrapper.

cont_cls¶

Container child class.

alias of BotoContainer

wrap_boto_errors = <cloud_browser.cloud.boto_base.BotoBucketWrapper object>¶: Exception translations.

class cloud_browser.cloud.boto_base.BotoContainer(conn, name=None, count=None, size=None)¶

Boto container wrapper.

classmethod from_bucket(connection, bucket)¶: Create from bucket object.

get_object(path)¶: Get single object.

get_objects(path, marker=None, limit=20)¶: Get objects.

max_list = 10000¶

Maximum number of objects that can be listed or None.

boto transparently pages through objects, so there is no real limit to the number of object that can be displayed. However, for practical reasons, we’ll limit it to the same as Rackspace.

obj_cls¶

Storage object child class.

alias of BotoObject

wrap_boto_errors = <cloud_browser.cloud.boto_base.BotoBucketWrapper object>¶: Exception translations.

class cloud_browser.cloud.boto_base.BotoExceptionWrapper¶

Boto boto exception translator.

error_cls¶: alias of cloud_browser.cloud.errors.CloudException

translate(exc)¶: Return whether or not to do translation.

class cloud_browser.cloud.boto_base.BotoKeyWrapper¶

Boto boto key exception translator.

error_cls¶: alias of cloud_browser.cloud.errors.NoObjectException

class cloud_browser.cloud.boto_base.BotoObject(container, name, **kwargs)¶

Boto ‘key’ object wrapper.

classmethod from_key(container, key)¶: Create from key object.

classmethod from_prefix(container, prefix)¶: Create from prefix object.

classmethod from_result(container, result)¶: Create from ambiguous result.

classmethod is_key(result)¶: Return True if result is a key object.

classmethod is_prefix(result)¶: Return True if result is a prefix object.

wrap_boto_errors = <cloud_browser.cloud.boto_base.BotoKeyWrapper object>¶: Exception translations.

Amazon Web Services S3 Datastore¶

Amazon Simple Storage Service (S3) datastore.

Note

Installation: Use of this module requires the open source boto package.

class cloud_browser.cloud.aws.AwsConnection(account, secret_key)¶

AWS connection wrapper.

cont_cls¶

Container child class.

alias of AwsContainer

class cloud_browser.cloud.aws.AwsContainer(conn, name=None, count=None, size=None)¶

AWS container wrapper.

obj_cls¶

Storage object child class.

alias of AwsObject

class cloud_browser.cloud.aws.AwsObject(container, name, **kwargs)¶

AWS ‘key’ object wrapper.

classmethod is_key(result)¶: Return True if result is a key object.

classmethod is_prefix(result)¶: Return True if result is a prefix object.

Google Storage Datastore¶

Google Storage for Developers datastore.

Google Storage for Developers (GS) is cosmetically an implementation of the S3 interface by Google.

Note

Installation: Use of this module requires the open source boto package. Not sure exactly which version installed GS support, but we’ll validate the version if it becomes an issue.

class cloud_browser.cloud.google.GsConnection(account, secret_key)¶

Google Storage connection wrapper.

cont_cls¶

Container child class.

alias of GsContainer

class cloud_browser.cloud.google.GsContainer(conn, name=None, count=None, size=None)¶

Google Storage container wrapper.

get_objects(path, marker=None, limit=20)¶

Get objects.

Certain upload clients may add a 0-byte object (e.g., FOLDER object for path path/to/FOLDER - path/to/FOLDER/FOLDER). We add an extra +1 limit query and ignore any such file objects.

obj_cls¶

Storage object child class.

alias of GsObject

class cloud_browser.cloud.google.GsObject(container, name, **kwargs)¶

Google Storage ‘key’ object wrapper.

classmethod from_prefix(container, prefix)¶: Create from prefix object.

classmethod is_key(result)¶: Return True if result is a key object.

classmethod is_prefix(result)¶: Return True if result is a prefix object.

Note

Boto uses the S3 Prefix object for GS prefixes.

Cloud / Datastore Support¶

Configuration¶

Errors¶

Datastores¶

Abstract Base Class¶

Filesystem Datastore¶

Apache Libcloud Datastore¶

Boto-based Datastore Abstract Base Class¶

Amazon Web Services S3 Datastore¶

Google Storage Datastore¶

Rackspace Cloud Files Datastore¶