ll.url – RFC 2396 compliant URLs

ll.url contains an RFC 2396 compliant implementation of URLs and classes for accessing resource metadata as well as file like classes for reading and writing resource data.

These three levels of functionality are implemented in three classes:

URL

URL objects are the names of resources and can be used and modified, regardless of the fact whether these resources actually exist. URL objects never hits the hard drive or the net.

Connection

Connection objects contain functionality that accesses and changes file metadata (like last modified date, permission bits, directory structure etc.). A connection object can be created by calling the connect() method on a URL object.

Resource

Resource objects are file like objects that work with the actual bytes that make up the file data. This functionality lives in the Resource class and its subclasses. Creating a resource is done by calling the open() method on a Connection or a URL.

Module documentation

ll.url.httpdate(dt)

Return a string suitable for a “Last-Modified” and “Expires” header.

dt is a datetime.datetime object in UTC.

class ll.url.Context

Bases: object

Working with URLs (e.g. calling URL.open() or URL.connect()) involves Connection objects. To avoid constantly creating new connections you can pass a Context object to those methods. Connections will be stored in the Context object and will be reused by those methods.

A Context object can also be used as a context manager. This context object will be used for all open() and connect() calls inside the with block. (Note that after the end of the with block all connections will be closed.)

closeall()

Close and drop all connections in this context.

class ll.url.Cursor

Bases: object

A Cursor object is used by the walk() method during directory traversal. It contains information about the state of the traversal and can be used to influence which directories are traversed and in which order.

Information about the state of the traversal is provided in the following attributes:

rooturl

The URL where traversal has been started (i.e. the object for which the walk() method has been called)

url

The current URL being traversed.

event

A string that specifies which event is currently handled. Possible values are: "beforedir", "afterdir" and "file". A "beforedir" event is emitted before a directory is entered. "afterdir" is emitted after a directory has been entered. "file" is emitted when a file is encountered.

isdir

True if url refers to a directory.

isfile

Tur if url refers to a regular file.

The following attributes specify which part of the tree should be traversed:

beforedir

Should the generator yield "beforedir" events?

afterdir

Should the generator yield "afterdir" events?

file

Should the generator yield "file" events?

enterdir

Should the directory be entered?

Note that if any of these attributes is changed by the code consuming the generator, this new value will be used for the next traversal step once the generator is resumed and will be reset to its initial value (specified in the constructor) afterwards.

__init__(url, beforedir=True, afterdir=False, file=True, enterdir=False)

Create a new Cursor object for a tree traversal rooted at the node node.

The arguments beforedir, afterdir, file and enterdir are used as the initial values for the attributes of the same name. (see the class docstring for info about their use).

restore()

Restore the attributes beforedir, afterdir, file and enterdir to their initial value.

class ll.url.Connection

Bases: object

A Connection object is used for accessing and modifying the metadata associated with a file. It is created by calling the connect() method on a URL object.

stat(url)

Return the result of a stat() call on the file url.

lstat(url)

Return the result of a stat() call on the file url. Like stat(), but does not follow symbolic links.

chmod(url, mode)

Set the access mode of the file url to mode.

chown(url, owner=None, group=None)

Change the owner and/or group of the file url.

lchown(url, owner=None, group=None)

Change the owner and/or group of the file url (ignoring symbolic links).

uid(url)

Return the user id of the owner of the file url.

gid(url)

Return the group id the file url belongs to.

owner(url)

Return the name of the owner of the file url.

group(url)

Return the name of the group the file url belongs to.

mimetype(url)

Return the mimetype of the file url.

exists(url)

Test whether the file url exists.

isfile(url)

Test whether the resource url is a file.

isdir(url)

Test whether the resource url is a directory.

islink(url)

Test whether the resource url is a link.

ismount(url)

Test whether the resource url is a mount point.

access(url, mode)

Test for access to the file/resource url.

size(url)

Return the size of the file url.

imagesize(url)

Return the size of the image url (if the resource is an image file) as a (width, height) tuple. This requires the PIL.

cdate(url)

Return the “metadate change” date of the file/resource url as a datetime.datetime object in UTC.

adate(url)

Return the last access date of the file/resource url as a datetime.datetime object in UTC.

mdate(url)

Return the last modification date of the file/resource url as a datetime.datetime object in UTC.

resheaders(url)

Return the MIME headers for the file/resource url.

remove(url)

Remove the file url.

rmdir(url)

Remove the directory url.

rename(url, target)

Renames url to target. This might not work if target has a different scheme than url (or is on a different server).

link(url, target)

Create a hard link from url to target. This will not work if target has a different scheme than url (or is on a different server).

symlink(url, target)

Create a symbolic link from url to target. This will not work if target has a different scheme than url (or is on a different server).

chdir(url)

Change the current directory to url.

mkdir(url, mode=511)

Create the directory url.

makedirs(url, mode=511)

Create the directory url and all intermediate ones.

walk(url, beforedir=True, afterdir=False, file=True, enterdir=True)

Return an iterator for traversing the directory hierarchy rooted at the directory url.

Each item produced by the iterator is a Cursor object. It contains information about the state of the traversal and can be used to influence which parts of the directory hierarchy are traversed and in which order.

The arguments beforedir, afterdir, file and enterdir specify how the directory hierarchy should be traversed. For more information see the Cursor class.

Note that the Cursor object is reused by walk(), so you can’t rely on any attributes remaining the same across calls to next().

The following example shows how to traverse the current directory, print all files except those in certain directories:

from ll import url

for cursor in url.here().walk(beforedir=True, afterdir=False, file=True):
        if cursor.isdir:
                if cursor.url.path[-2] in (".git", "build", "dist", "__pycache__"):
                        cursor.enterdir = False
        else:
                print(cursor.url)

listdir(url, include=None, exclude=None, ignorecase=False)

Iterates over items in the directory url. The items produced are URL objects relative to url.

With the optional include argument, this only lists items whose names match the given pattern. Items matching the optional pattern exclude will not be listed. include and exclude can be strings (which will be interpreted as fnmatch style filename patterns) or lists of strings. If ignorecase is true case-insensitive name matching will be performed.

files(url, include=None, exclude=None, ignorecase=False)

Iterates over files in the directory url. The items produced are URL objects relative to url.

With the optional include argument, this only lists files whose names match the given pattern. Files matching the optional pattern exclude will not be listed. include and exclude can be strings (which will be interpreted as fnmatch style filename patterns) or lists of strings. If ignorecase is true case-insensitive name matching will be performed.

dirs(url, include=None, exclude=None, ignorecase=False)

Iterates over directories in the directory url. The items produced are URL objects relative to url.

With the optional include argument, this only directories items whose names match the given pattern. Directories matching the optional pattern exclude will not be listed. include and exclude can be strings (which will be interpreted as fnmatch style filename patterns) or lists of strings. If ignorecase is true case-insensitive name matching will be performed.

walkall(url, include=None, exclude=None, enterdir=None, skipdir=None, ignorecase=False)

Recursively iterate over files and subdirectories. The iterator yields URL objects naming each child URL of the directory url and its descendants relative to url. This performs a depth-first traversal, returning each directory before all its children.

With the optional include argument, only yield items whose names match the given pattern. Items matching the optional pattern exclude will not be listed. Directories that don’t match the optional pattern enterdir or match the pattern skipdir will not be traversed. include, exclude, enterdir and skipdir can be strings (which will be interpreted as fnmatch style filename patterns) or lists of strings. If ignorecase is true case-insensitive name matching will be performed.

walkfiles(url, include=None, exclude=None, enterdir=None, skipdir=None, ignorecase=False)

Return a recursive iterator over files in the directory url.

With the optional include argument, only yield files whose names match the given pattern. Files matching the optional pattern exclude will not be listed. Directories that don’t match the optional pattern enterdir or match the pattern skipdir will not be traversed. include, exclude, enterdir and skipdir can be strings (which will be interpreted as fnmatch style filename patterns) or lists of strings. If ignorecase is true case-insensitive name matching will be performed.

walkdirs(url, include=None, exclude=None, enterdir=None, skipdir=None, ignorecase=False)

Return a recursive iterator over subdirectories in the directory url.

With the optional include argument, only yield directories whose names match the given pattern. Items matching the optional pattern exclude will not be listed. Directories that don’t match the optional pattern enterdir or match the pattern skipdir will not be traversed. include, exclude, enterdir and skipdir can be strings (which will be interpreted as fnmatch style filename patterns) or lists of strings. If ignorecase is true case-insensitive name matching will be performed.

open(url, *args, **kwargs)

Open url for reading or writing. open() returns a Resource object.

Which additional parameters are supported depends on the actual resource created. Some common parameters are:

modestr

A string indicating how the file is to be opened (just like the mode argument for the builtin open() (e.g. "rb" or "wb").

headersdict

Additional headers to use for an HTTP request.

databytes

Request body to use for an HTTP POST request.

pythonstr or None

Name of the Python interpreter to use on the remote side (used by ssh URLs)

niceint or None

Nice level for the remote python (used by ssh URLs)

checkbool or None

Whether ssh host keys should be checked (used by ssh URLs where it defaults to True and ssh-nocheck URLs where it defaults to False).

class ll.url.LocalConnection

Bases: Connection

A LocalConnection object is used for accessing and modifying the metadata associated with a file in the local filesystem. It is created by calling the connect() method on a URL object with no scheme or the file or root scheme.

class ll.url.SshConnection

Bases: Connection

A SshConnection object is used for accessing and modifying the metadata associated with a file on a remote filesystem. Remote files will be accessed via code executed remotely on the target host via execnet.

SshConnection objects are created by calling the connect() method on a URL object with the ssh or ssh-nocheck scheme.

Note

Using the scheme ssh-nocheck disables checks of the host key, i.e. it passes -o "StrictHostKeyChecking=no" to the underlying ssh command.

If you need to use further options (e.g. when your known_hosts file isn’t writable), you should configure that in your ~/.ssh/config file, for example:

Host foo
        Hostname foo.example.org
        StrictHostKeyChecking no
        UserKnownHostsfile /dev/null

or for Windows:

Host foo
        Hostname foo.example.org
        StrictHostKeyChecking no
        UserKnownHostsfile nul:

class ll.url.URLConnection

Bases: Connection

A URLConnection object is used for accessing and modifying the metadata associated any other resource specified by a URL (except those handled by the other Connection subclasses).

ll.url.here(scheme='file')

Return the current directory as an URL object.

ll.url.home(user='', scheme='file')

Return the home directory of the current user (or the user named user, if user is specified) as an URL object:

>>> url.home()
URL('file:/home/walter/')
>>> url.home("andreas")
URL('file:/home/andreas/')

ll.url.root()

Return a blank root URL, i.e. URL("root:").

ll.url.File(name, scheme='file')

Turn a filename into an URL object:

>>> url.File("a#b")
URL('file:a%23b')

ll.url.Dir(name, scheme='file')

Turns a directory name into an URL object, just like File(), but ensures that the path is terminated with a /:

>>> url.Dir("a#b")
URL('file:a%23b/')

ll.url.Ssh(user, host, path='~/')

Return a ssh URL for the user user on the host host with the path path. path (defaulting to the users home directory) must be a path in URL notation (i.e. use / as directory separator):

>>> url.Ssh("root", "www.example.com", "~joe/public_html/index.html")
URL('ssh://root@www.example.com/~joe/public_html/index.html')

If the path starts with ~/ it is relative to this users home directory, if it starts with ~user it’s relative to the home directory of the user user. In all other cases the path is considered to be absolute.

ll.url.first(urls)

Return the first URL from urls that exists as a real file or directory. None entries in urls will be skipped.

ll.url.firstdir(urls)

Return the first URL from urls that exists as a real directory. None entries in urls will be skipped.

ll.url.firstfile(urls)

Return the first URL from urls that exists as a real file. None entries in urls will be skipped.

class ll.url.Resource

Bases: object

A Resource is a base class that provides a file-like interface to local and remote files, URLs and other resources.

Each resource object has the following attributes:

url

The URL for which this resource has been opened (i.e. foo.open().url is foo if foo is a URL object);

name

A string version of url;

closed

A bool specifying whether the resource has been closed (i.e. whether the close() method has been called).

In addition to file methods (like read(), readlines(), write() and close()) a resource object might provide the following methods:

finalurl()

Return the real URL of the resource (this might be different from the url attribute in case of a redirect).

size()

Return the size of the file/resource.

mdate()

Return the last modification date of the file/resource as a datetime.datetime object in UTC.

mimetype()

Return the mimetype of the file/resource.

imagesize()

Return the size of the image (if the resource is an image file) as a (width, height) tuple. This requires the PIL.

class ll.url.FileResource

Bases: Resource

A subclass of Resource that handles local files.

class ll.url.RemoteFileResource

Bases: Resource

A subclass of Resource that handles remote files (i.e. those using the ssh scheme).

class ll.url.URLResource

Bases: Resource

A subclass of Resource that handles HTTP, FTP and other URLs (i.e. those that are not handled by FileResource or RemoteFileResource.

class ll.url.SchemeDefinition

Bases: object

A SchemeDefinition instance defines the properties of a particular URL scheme.

__init__(scheme, usehierarchy, useserver, usefrag, islocal=False, isremote=False, defaultport=None)

Create a new SchemeDefinition instance. Arguments are:

• scheme: The name of the scheme;

• usehierarchy: Specifies whether this scheme uses hierarchical URLs or opaque URLs (i.e. whether hier_part or opaque_part from the BNF in RFC 2396 is used);

• useserver: Specifies whether this scheme uses an Internet-based server authority component or a registry of naming authorities (only for hierarchical URLs);

• usefrag: Specifies whether this scheme uses fragments (according to the BNF in RFC 2396 every scheme does, but it doesn’t make sense for e.g. "javascript", "mailto" or "tel");

• islocal: Specifies whether URLs with this scheme refer to local files;

• isremote: Specifies whether URLs with this scheme refer to remote files (there may be schemes which are neither local nor remote, e.g. "mailto");

• defaultport: The default port for this scheme (only for schemes using server based authority).

connect(url, context=None, **kwargs)

Create a Connection for the URL url (which must have self as the scheme).

closeall(context)

Close all connections active for this scheme in the context context.

class ll.url.URL

Bases: object

An RFC 2396 compliant URL.

__init__(url=None)

Create a new URL instance. url may be a str object, or an URL (in which case you’ll get a copy of url), or None (which will create an URL referring to the “current document”).

clone()

Return an identical copy self.

withext(ext)

Return a new URL where the filename extension has been replaced with ext.

withoutext()

Return a new URL where the filename extension has been removed.

withfile(file)

Return a new URL where the filename (i.e. the name of last component of path_segments) has been replaced with file.

withfrag(frag)

Return a new URL where the fragment has been replaced with frag.

withoutfrag()

Return a new URL where the frag has been dropped.

__truediv__(other)

Join self with another (possible relative) URL other, to form a new URL.

other may be a str or URL object. It may be None (referring to the “current document”) in which case self will be returned. It may also be a list or other iterable. For this case a list (or iterator) will be returned where __div__() will be applied to every item in the list/iterator. E.g. the following expression returns all the files in the current directory as absolute URLs (see the method files() and the function here() for further explanations):

>>> here = url.here()
>>> for f in here/here.files():
...     print(f)

__rtruediv__(other)

Right hand version of __div__(). This supports lists and iterables as the left hand side too.

relative(baseurl, allowschemerel=False)

Return an relative URL rel such that baseurl/rel == self, i.e. this is the inverse operation of __div__().

If self is relative, has a different scheme or authority than baseurl or a non-hierarchical scheme, an identical copy of self will be returned.

If allowschemerel is true, scheme relative URLs are allowed, i.e. if both self and baseurl use the same hierarchical scheme, but a different authority (i.e. server), a scheme relative url (//server/path/file.html) will be returned.

__bool__()

Return whether the URL is not empty, i.e. whether it is not the URL referring to the start of the current document.

__eq__(other)

Return whether two URL objects are equal. Note that only properties relevant for the current scheme will be compared.

__ne__(other)

Return whether two URL objects are not equal.

__hash__()

Return a hash value for self, to be able to use URL objects as dictionary keys. You must be careful not to modify an URL as soon as you use it as a dictionary key.

abs(scheme=-1)

Return an absolute version of self (works only for local URLs).

If the argument scheme is specified, it will be used for the resulting URL otherwise the result will have the same scheme as self.

real(scheme=-1)

Return the canonical version of self, eliminating all symbolic links (works only for local URLs).

If the argument scheme is specified, it will be used for the resulting URL otherwise the result will have the same scheme as self.

islocal()

Return whether self refers to a local file, i.e. whether self is a relative URL or the scheme is root or file).

local()

Return self as a local filename (which will only works if self is local (see islocal()).

connect(context=None, **kwargs)

Return a Connection object for accessing and modifying the metadata of self.

Whether you get a new connection object, or an existing one depends on the scheme, the URL itself, and the context passed in (as the context argument).

open(*args, **kwargs)

Open self for reading or writing. open() returns a Resource object.

Which additional parameters are supported depends on the actual resource created. Some common parameters are:

mode (supported by all resources)

A string indicating how the file is to be opened (just like the mode argument for the builtin open(); e.g. "rb" or "wb").

context (supported by all resources)

open() needs a Connection for this URL which it gets from a Context object.

headers

Additional headers to use for an HTTP request.

data

Request body to use for an HTTP POST request.

python

Name of the Python interpreter to use on the remote side (used by ssh URLs)

nice

Nice level for the remove python (used by ssh URLs)

import_(name=None)

Import the content of the URL self as a Python module.

name can be used the specify the module name (i.e. the __name__ attribute of the module). The default determines it from the URL.

Special features of ll.url

The class ll.url.URL supports many common schemes and one additional special scheme named root that deserves an explanation.

A root URL is supposed to be an URL that is relative to a “project” directory instead to a base URL of the document that contains the URL.

Suppose we have a document with the following base URL:

>>> from ll import url
>>> base = url.URL("root:company/it/about/index.html")

Now, if we have the following relative URL in this document:

>>> url1 = url.URL("images/logos/spam.png")

the combined URL will be:

>>> base/url1
URL('root:company/it/about/images/logos/spam.png')

Now it we use this combined URL and interpret it relative to the base URL we get back our original relative URL:

>>> (base/url1).relative(base)
URL('images/logos/spam.png')

Let’s try a root URL now:

>>> url2 = url.URL("root:images/logos/spam.png")

Combining this URL with the base URL gives us the same as url2:

>>> base/url2
URL('root:images/logos/spam.png')

But if we interpret this result relative to base, we’ll get:

>>> (base/url2).relative(base)
URL('../../../images/logos/spam.png')

I.e. this gives us a relative URL that references url2 from base when both URLs are relative to the same root directory.