parse – Parsing XML/HTML

This module contains everything you need to create XIST objects by parsing files, strings, URLs etc.

Parsing XML is done with a pipelined approach. The first step in the pipeline is a source object that provides the input for the rest of the pipeline. The next step is the XML parser. It turns the input source into an iterator over parsing events (an “event stream”). Further steps in the pipeline might resolve namespace prefixes (NS), and instantiate XIST classes (Node). The final step in the pipeline is either building an XML tree via tree() or an iterative parsing step (similar to ElementTrees iterparse() function) via itertree().

Parsing a simple HTML string might e.g. look like this:

>>> from ll.xist import xsc, parse
>>> from ll.xist.ns import html
>>> source = b"<a href='http://www.python.org/'>Python</a>"
>>> doc = parse.tree(
...     parse.String(source),
...     parse.Expat(),
...     parse.NS(html),
...     parse.Node(pool=xsc.Pool(html)),
... )
>>> doc.string()
'<a href="http://www.python.org/">Python</a>'

A source object is an iterable object that produces the input byte string for the parser (possibly in multiple chunks) (and information about the URL of the input):

>>> from ll.xist import parse
>>> list(parse.String(b"<a href='http://www.python.org/'>Python</a>"))
[('url', URL('STRING')),
 ('bytes', "<a href='http://www.python.org/'>Python</a>")]

All subsequent objects in the pipeline are callable objects, get the input iterator as an argument and return an iterator over events themselves. The following code shows an example of an event stream:

>>> from ll.xist import parse
>>> source = b"<a href='http://www.python.org/'>Python</a>"
>>> list(parse.events(parse.String(source), parse.Expat()))
[('url', URL('STRING')),
 ('position', (0, 0)),
 ('enterstarttag', 'a'),
 ('enterattr', 'href'),
 ('text', 'http://www.python.org/'),
 ('leaveattr', 'href'),
 ('leavestarttag', 'a'),
 ('position', (0, 39)),
 ('text', 'Python'),
 ('endtag', 'a')]

An event is a tuple consisting of the event type and the event data. Different stages in the pipeline produce different event types. The following event types can be produced by source objects:

"url"

The event data is the URL of the source. Usually such an event is produced only once at the start of the event stream. For sources that have no natural URL (like strings or streams) the URL can be specified when creating the source object.

"bytes"

This event is produced by source objects (and Transcoder objects). The event data is a byte string.

"str"

The event data is a string. This event is produced by Decoder or source objects. Note that the only predefined pipeline objects that can handle "str" events are Encoder objects, i.e. normally a parser handles "bytes" events, but not "str" events.

The following type of events are produced by parsers (in addition to the "url" event from above):

"position"

The event data is a tuple containing the line and column number in the source (both starting with 0). All the following events should use this position information until the next position event.

"xmldecl"

The XML declaration. The event data is a dictionary containing the keys "version", "encoding" and "standalone". Parsers may omit this event.

"begindoctype"

The begin of the doctype. The event data is a dictionary containing the keys "name", "publicid" and "systemid". Parsers may omit this event.

"enddoctype"

The end of the doctype. The event data is None. (If there is no internal subset, the "enddoctype" event immediately follows the "begindoctype" event). Parsers may omit this event.

"comment"

A comment. The event data is the content of the comment.

"text"

Text data. The event data is the text content. Parsers should try to avoid outputting multiple text events in sequence.

"cdata"

A CDATA section. The event data is the content of the CDATA section. Parsers may report CDATA sections as "text" events instead of "cdata" events.

"enterstarttag"

The beginning of an element start tag. The event data is the element name.

"leavestarttag"

The end of an element start tag. The event data is the element name. The parser will output events for the attributes between the "enterstarttag" and the "leavestarttag" event.

"enterattr"

The beginning of an attribute. The event data is the attribute name.

"leaveattr"

The end of an attribute. The event data is the attribute name. The parser will output events for the attribute value between the "enterattr" and the "leaveattr" event. (In almost all cases this is one text event).

"endtag"

An element end tag. The event data is the element name.

"procinst"

A processing instruction. The event data is a tuple consisting of the processing instruction target and the data.

"entity"

An entity reference. The event data is the entity name.

The following events are produced for elements and attributes in namespace mode (instead of those without the ns suffix). They are produced by NS objects or by Expat objects when the ns argument is true (i.e. the expat parser performs the namespace resolution):

"enterstarttagns"

The beginning of an element start tag in namespace mode. The event data is an (namespace name, element name) tuple.

"leavestarttagns"

The end of an element start tag in namespace mode. The event data is an (namespace name, element name) tuple.

"enterattrns"

The beginning of an attribute in namespace mode. The event data is an (namespace name, element name) tuple.

"leaveattrns"

The end of an attribute in namespace mode. The event data is an (namespace name, element name) tuple.

"endtagns"

An element end tag in namespace mode. The event data is an (namespace name, element name) tuple.

Once XIST nodes have been instantiated (by Node objects) the following events are used:

"xmldeclnode"

The XML declaration. The event data is an instance of ll.xist.ns.xml.XML.

"doctypenode"

The doctype. The event data is an instance of ll.xist.xsc.DocType.

"commentnode"

A comment. The event data is an instance of ll.xist.xsc.Comment.

"textnode"

Text data. The event data is an instance of ll.xist.xsc.Text.

"enterelementnode"

The beginning of an element. The event data is an instance of ll.xist.xsc.Element (or one of its subclasses). The attributes of the element object are set, but the element has no content yet.

"leaveelementnode"

The end of an element. The event data is an instance of ll.xist.xsc.Element.

"procinstnode"

A processing instruction. The event data is an instance of ll.xist.xsc.ProcInst.

"entitynode"

An entity reference. The event data is an instance of ll.xist.xsc.Entity.

For consuming event streams there are three functions:

events()

This generator simply outputs the events.

tree()

This function builds an XML tree from the events and returns it.

itertree()

This generator builds a tree like tree(), but returns events during certain steps in the parsing process.

Example

The following example shows a custom generator in the pipeline that lowercases all element and attribute names:

from ll.xist import xsc, parse
from ll.xist.ns import html

def lowertag(input):
        for (event, data) in input:
                if event in {"enterstarttag", "leavestarttag", "endtag", "enterattr", "leaveattr"}:
                        data = data.lower()
                yield (event, data)

e = parse.tree(
        parse.String(b"<A HREF='gurk'><B>gurk</B></A>"),
        parse.Expat(),
        lowertag,
        parse.NS(html),
        parse.Node(pool=xsc.Pool(html))
)

print(e.string())

This scripts outputs:

<a href="gurk"><b>gurk</b></a>

exception ll.xist.parse.UnknownEventError

Bases: TypeError

This exception is raised when a pipeline object doesn’t know how to handle an event.

class ll.xist.parse.String

Bases: object

Provides parser input from a string.

__init__(data, url=None)

Create a String object. data must be a bytes or str object. url specifies the URL for the source (defaulting to "STRING").

__iter__()

Produces an event stream of one "url" event and one "bytes" or "str" event for the data.

class ll.xist.parse.Iter

Bases: object

Provides parser input from an iterator over strings.

__init__(iterable, url=None)

Create a Iter object. iterable must be an iterable object producing bytes or str objects. url specifies the URL for the source (defaulting to "ITER").

__iter__()

Produces an event stream of one "url" event followed by the "bytes"/"str" events for the data from the iterable.

class ll.xist.parse.Stream

Bases: object

Provides parser input from a stream (i.e. an object that provides a read() method).

__init__(stream, url=None, bufsize=8192)

Create a Stream object. stream must have a read() method (with a size argument). url specifies the URL for the source (defaulting to "STREAM"). bufsize specifies the chunksize for reads from the stream.

__iter__()

Produces an event stream of one "url" event followed by the "bytes"/"str" events for the data from the stream.

class ll.xist.parse.File

Bases: object

Provides parser input from a file.

__init__(filename, bufsize=8192)

Create a File object. filename is the name of the file and may start with ~ or ~user for the home directory of the current or the specified user. bufsize specifies the chunksize for reads from the file.

__iter__()

Produces an event stream of one "url" event followed by the "bytes" events for the data from the file.

class ll.xist.parse.URL

Bases: object

Provides parser input from a URL.

__init__(name, bufsize=8192, *args, **kwargs)

Create a URL object. name is the URL. bufsize specifies the chunksize for reads from the URL. args and kwargs will be passed on to the open() method of the URL object.

The URL for the input will be the final URL for the resource (i.e. it will include redirects).

__iter__()

Produces an event stream of one "url" event followed by the "bytes" events for the data from the URL.

class ll.xist.parse.ETree

Bases: object

Produces a (namespaced) event stream from an object that supports the ElementTree API.

__init__(data, url=None, defaultxmlns=None)

Create an ETree object. Arguments have the following meaning:

data

An object that supports the ElementTree API.

url

The URL of the source. Defaults to "ETREE".

defaultxmlns

The namespace name (or a namespace module containing a namespace name) that will be used for all elements that don’t have a namespace.

__iter__()

Produces an event stream of namespaced parsing events for the ElementTree object passed as data to the constructor.

class ll.xist.parse.Decoder

Bases: object

Decode the bytes object produced by the previous object in the pipeline to str object.

This input object can be a source object or any other pipeline object that produces bytes objects.

__init__(encoding=None)

Create a Decoder object. encoding is the encoding of the input. If encoding is None it will be automatically detected from the XML data.

class ll.xist.parse.Encoder

Bases: object

Encode the str objects produced by the previous object in the pipeline to bytes objects.

This input object must be a pipeline object that produces string output (e.g. a Decoder object).

This can e.g. be used to parse a str object instead of a bytes object like this:

>>> from ll.xist import xsc, parse
>>> from ll.xist.ns import html
>>> source = "<a href='http://www.python.org/'>Python</a>"
>>> doc = parse.tree(
...     parse.String(source),
...     parse.Encoder(encoding="utf-8"),
...     parse.Expat(encoding="utf-8"),
...     parse.NS(html),
...     parse.Node(pool=xsc.Pool(html)),
... )
>>> doc.string()
'<a href="http://www.python.org/">Python</a>'

__init__(encoding=None)

Create an Encoder object. encoding will be the encoding of the output. If encoding is None it will be automatically detected from the XML declaration in the data.

class ll.xist.parse.Transcoder

Bases: object

Transcode the bytes object of the input object into another encoding.

This input object can be a source object or any other pipeline object that produces bytes events.

__init__(fromencoding=None, toencoding=None)

Create a Transcoder object. fromencoding is the encoding of the input. toencoding is the encoding of the output. If any of them is None the encoding will be detected from the data.

class ll.xist.parse.Parser

Bases: object

Basic parser interface.

class ll.xist.parse.Expat

Bases: Parser

A parser using Pythons builtin expat parser.

__init__(encoding=None, xmldecl=False, doctype=False, loc=True, cdata=False, ns=False)

Create an Expat parser. Arguments have the following meaning:

encodingstring or None

Forces the parser to use the specified encoding. The default None results in the encoding being detected from the XML itself.

xmldeclbool

Should the parser produce events for the XML declaration?

doctypebool

Should the parser produce events for the document type?

locbool

Should the parser produce "location" events?

cdatabool

Should the parser output CDATA sections as "cdata" events? (If cdata is false "text" events are output instead.)

nsbool

If ns is true, the parser performs namespace processing itself, i.e. it will emit "enterstarttagns", "leavestarttagns", "endtagns", "enterattrns" and "leaveattrns" events instead of "enterstarttag", "leavestarttag", "endtag", "enterattr" and "leaveattr" events.

__call__(input)

Return an iterator over the events produced by input.

class ll.xist.parse.SGMLOP

Bases: Parser

A parser based on sgmlop.

__init__(encoding=None, cdata=False)

Create a SGMLOP parser. Arguments have the following meaning:

encodingstring or None

Forces the parser to use the specified encoding. The default None results in the encoding being detected from the XML itself.

cdatabool

Should the parser output CDATA sections as "cdata" events? (If cdata is false output "text" events instead.)

__call__(input)

Return an iterator over the events produced by input.

class ll.xist.parse.NS

Bases: object

An NS object is used in a parsing pipeline to add support for XML namespaces. It replaces the "enterstarttag", "leavestarttag", "endtag", "enterattr" and "leaveattr" events with the appropriate namespace version of the events (i.e. "enterstarttagns" etc.) where the event data is a (namespace, name) tuple.

The output of an NS object in the stream looks like this:

>>> from ll.xist import parse
>>> from ll.xist.ns import html
>>> list(parse.events(
...     parse.String(b"<a href='http://www.python.org/'>Python</a>"),
...     parse.Expat(),
...     parse.NS(html)
... ))
[('url', URL('STRING')),
 ('position', (0, 0)),
 ('enterstarttagns', ('http://www.w3.org/1999/xhtml', 'a')),
 ('enterattrns', (None, 'href')),
 ('text', 'http://www.python.org/'),
 ('leaveattrns', (None, 'href')),
 ('leavestarttagns', ('http://www.w3.org/1999/xhtml', 'a')),
 ('position', (0, 39)),
 ('text', 'Python'),
 ('endtagns', ('http://www.w3.org/1999/xhtml', 'a'))]

__init__(prefixes=None, **kwargs)

Create an NS object. prefixes (if not None) can be a namespace name (or module), which will be used for the empty prefix, or a dictionary that maps prefixes to namespace names (or modules). kwargs maps prefixes to namespaces names too. If a prefix is in both prefixes and kwargs, kwargs wins.

class ll.xist.parse.Node

Bases: object

A Node object is used in a parsing pipeline to instantiate XIST nodes. It consumes a namespaced event stream:

>>> from ll.xist import xsc, parse
>>> from ll.xist.ns import html
>>> list(parse.events(
...     parse.String(b"<a href='http://www.python.org/'>Python</a>"),
...     parse.Expat(),
...     parse.NS(html),
...     parse.Node(pool=xsc.Pool(html))
... ))
[('enterelementnode',
  <element ll.xist.ns.html.a xmlns='http://www.w3.org/1999/xhtml' (no children/1 attr) location='STRING:0:0' at 0x10a683550>),
 ('textnode',
  <ll.xist.xsc.Text content='Python' location='STRING:0:39' at 0x10a5e1170>),
 ('leaveelementnode',
  <element ll.xist.ns.html.a xmlns='http://www.w3.org/1999/xhtml' (no children/1 attr) location='STRING:0:0' at 0x10a683550>)
]

The event data of all events are XIST nodes. The element node from the "enterelementnode" event already has all attributes set. There will be no events for attributes.

__init__(pool=None, base=None, loc=True)

Create a Node object.

pool may be None or a xsc.Pool object and specifies which classes used for creating element, entity and processsing instruction instances.

base specifies the base URL for interpreting relative links in the input.

loc specified whether location information should be attached to the nodes that get generated (the startloc attribute (and endloc attribute for elements))

class ll.xist.parse.Tidy

Bases: object

A Tidy object parses (potentially ill-formed) HTML from a source into a (non-namespaced) event stream by using lxml’s HTML parser:

>>> from ll.xist import parse
>>> list(parse.events(parse.URL("http://www.yahoo.com/"), parse.Tidy()))
[('url', URL('http://de.yahoo.com/?p=us')),
 ('enterstarttag', 'html'),
 ('enterattr', 'class'),
 ('text', 'y-fp-bg y-fp-pg-grad  bkt708'),
 ('leaveattr', 'class'),
 ('enterattr', 'lang'),
 ('text', 'de-DE'),
 ('leaveattr', 'lang'),
 ('enterattr', 'style'),
 ('leaveattr', 'style'),
 ('leavestarttag', 'html'),
...

__init__(encoding=None, xmldecl=False, doctype=False)

Create a new Tidy object. Parameters have the following meaning:

encodingstring or None

The encoding of the input. If encoding is None it will be automatically detected by the HTML parser.

xmldeclbool

Should the parser produce events for the XML declaration?

doctypebool

Should the parser produce events for the document type?

ll.xist.parse.events(*pipeline)

Return an iterator over the events produced by the pipeline objects in pipeline.

ll.xist.parse.tree(*pipeline, validate=False)

Return a tree of XIST nodes from the event stream pipeline.

pipeline must output only events that contain XIST nodes, i.e. the event types "xmldeclnode", "doctypenode", "commentnode", "textnode", "enterelementnode", "leaveelementnode", "procinstnode" and "entitynode".

If validate is true, the tree is validated, i.e. it is checked if the structure of the tree is valid (according to the model attribute of each element node), if no undeclared elements or attributes have been encountered, all required attributes are specified and all attributes have allowed values.

The node returned from tree() will always be a Frag object.

Example:

>>> from ll.xist import xsc, parse
>>> from ll.xist.ns import xml, html, chars
>>> doc = parse.tree(
...     parse.URL("http://www.python.org/"),
...     parse.Tidy(),
...     parse.NS(html),
...     parse.Node(pool=xsc.Pool(xml, html, chars))
... )

>>> doc[0]
<element ll.xist.ns.html.html
        xmlns='http://www.w3.org/1999/xhtml'
        (7 children/3 attrs)
        location='https://www.python.org/:?:?'
        at 0x110a4ecd0>

ll.xist.parse.itertree(*pipeline, entercontent=True, enterattrs=False, enterattr=False, enterelementnode=False, leaveelementnode=True, enterattrnode=True, leaveattrnode=False, selector=None, validate=False)

Parse the event stream pipeline iteratively.

itertree() still builds a tree, but it returns an iterator of xsc.Cursor objects that tracks changes to the tree as it is built.

validate specifies whether each node should be validated after it has been fully parsed.

The rest of the arguments can be used to control when itertree() returns to the calling code. For an explanation of their meaning see the class ll.xist.xsc.Cursor.

Example:

>>> from ll.xist import xsc, parse
>>> from ll.xist.ns import xml, html, chars
>>> for c in parse.itertree(
...     parse.URL("http://www.python.org/"),
...     parse.Tidy(),
...     parse.NS(html),
...     parse.Node(pool=xsc.Pool(xml, html, chars)),
...     selector=html.a/html.img
... ):
...     print(c.path[-1].attrs.src, "-->", c.path[-2].attrs.href)

https://www.python.org/static/img/python-logo.png --> https://www.python.org/