Data¶

Data is a fundamental HQL type for structured information.

It is not HQL's equivalent of a struct, record, object, or database row. Those abstractions describe values according to a predefined set of fields. Data instead describes a tree whose structure is carried by the value itself.

The closest existing representation is YAML. Conceptually, an HQL Data value should be able to represent a YAML document essentially one-to-one.

name: Alice
age: 42

address:
  city: Zürich
  country: CH

roles:
  - admin
  - author

This is naturally a Data tree:

Data
└── mapping
    ├── name
    │   └── "Alice"
    ├── age
    │   └── 42
    ├── address
    │   └── mapping
    │       ├── city
    │       │   └── "Zürich"
    │       └── country
    │           └── "CH"
    └── roles
        └── sequence
            ├── "admin"
            └── "author"

Data is recursive, and its declaration in std/core.hql says so directly:

type Data = union {
    Scalar,
    Seq<Data>,
    Map<String, Data>
}

A tree is a leaf, a sequence of trees, or a string-keyed map of trees, and nothing else. That is the whole of YAML's and JSON's data model, which is why Data is the natural type for querying either.

Data is a union¶

A supertype set, type X : {A, B}, says that X is every one of its parents at once. Data is the opposite situation: a value is one of the alternatives, never all three. That is what a union is for, and Data is where HQL genuinely needs one — see type-system.

A type is a Data by being one of its members, not by declaring Data as a parent:

count : Data = 3                           // a leaf
hours : Data = ["dusk", "dawn"]            // a sequence of trees
tally : Data = Map(["barn-owl"], [2])      // a string-keyed map of trees
note  : Data = { species: "barn-owl" }     // a tree literal

Membership recurses through the argument, so a List<List<Map<String, Float>>> is a tree because each layer is. Two things that look close are not trees, and the members say why: a Set<Int> has no positions, and a tree's children have them; a Map<Int, Int> has no string keys.

Trees of different types are held together as Data. Int and String share no parent a list could be held at, and both are trees, so [1, "owl"] is a List<Data> — exactly what it is in JSON — and { tags: [1, "owl", [true, 2.5]] } is a tree literal like any other. Declared ancestry is tried first, so [1, 2] stays a List<Int>. A card is not a tree, and [1, [[owl]]] has no common type.

Membership runs one way. Knowing a value is Data does not make it a Scalar or a Seq<Data>; finding out which it is belongs to refinement.

Viewing a value as Data never converts it. tally above is still the Map<String, Int> it was, and only what the checker knows about it has changed. This is the same rule refinement follows in the other direction.

Scalars are the leaves¶

abstract type Scalar

type Bool   : Scalar
type Int    : {Scalar, Orderable}
type Float  : {Scalar, Orderable}
type String : {Scalar, Orderable}
type Date   : {Scalar, Orderable}

Scalar is an abstract type of law: a value with no parts of its own. Being a leaf and having an order are separate contracts, which is why they are separate parents. Bool is a leaf with no order worth sorting by. A Card has an order, by its name, and is no leaf. The four ordered scalars satisfy both, and {Scalar, Orderable} is an ordinary supertype set.

Date is declared so a schema can name it. No literal produces one yet, and its operations remain open.

Metadata is Data¶

One of the main purposes of Data is metadata.

Metadata usually does not deserve a new HQL record type merely because it has several named properties. Its natural representation is a Data tree.

A document header is the canonical example. Every Doc — and so every card, which is a narrowed document — carries a header that is a Data tree.

A card may begin with optional YAML metadata:

---
title: Data
tags: [hql, type-system]
status: draft

source:
  format: yaml
  version: 1
---

The complete header is Data.

Conceptually:

doc.header : Data

header is the name, because a fenced YAML block is only a Markdown-specific way of writing the authored entries of a header, and a header also carries a title the host derived when nobody wrote one. The header is not optional: a document with no metadata block has an empty header. See doc-type.

The metadata can be traversed naturally:

doc.header.title
doc.header.tags
doc.header.status

A single open tree is what lets a derived title sit beside authored keys without a second metadata type. Where a document is — name, path, format — is not in the tree at all but a field of Doc, so an authored key of that name contradicts nothing, and an extension's contributions are assembled into card.metadata rather than merged in here.

These expressions do not imply that HQL contains a statically declared header struct with fields called title, tags, status, and source.

They are paths through a Data tree.

This distinction is central because a document header is deliberately extensible. HMD reserves only tags, use, import, and nav in the authored block; every other key belongs to the author, and the derived and contributed layers add keys no author wrote. Different cards, applications, extensions, and namespaces can place different metadata there without extending the HQL type system every time a new YAML key appears.

The same principle applies beyond document headers. Configuration, annotations, extension metadata, query metadata, imported YAML, JSON-like API responses, database JSON values, and similar structured payloads are naturally Data.

In other words, when HQL encounters structured metadata for which there is no stronger semantic type, the default type is Data.

Data is not a record¶

A record describes a known shape.

For example:

type Person {
    name: String
    age: Int
}

A value of Person is constrained by the definition of Person.

Data has no such requirement.

name: Alice
age: 42

and:

server:
  host: localhost
  ports:
    - 8080
    - 8443

are both perfectly valid Data values even though their shapes have nothing in common.

A record therefore answers:

What is the declared shape of this value?

Data answers:

What tree is present here?

This also distinguishes Data from a database row. A row belongs to a relation with columns and a schema. Data is hierarchical and carries arbitrary nesting.

Structure first, stronger types later¶

Representing something as Data does not prevent HQL from giving parts of it stronger types.

A tree can be interpreted, validated, or converted later.

Conceptually:

metadata : Data = ...

person : Person =
    metadata.as<Person>()

This gives HQL a useful separation between raw structured information and typed interpretation. Interpretation need not copy: HMD, extensions and the program itself progressively establish stronger types for regions of one unchanged tree. See type-system.

The source can remain faithful to YAML while an extension, function, schema, or application imposes additional meaning where necessary.

The same idea applies to HyperMarkDown itself. The entire header remains Data, while HMD or HQL can assign special meaning to particular entries.

The tree does not have to stop being Data merely because some of its paths are understood by the system.

This is what makes card types narrow rather than being declared up front. See card-type.

YAML as the native serialization of Data¶

YAML provides a useful way of thinking about the external representation of Data.

Loading YAML should naturally produce Data:

config : Data = yaml.load("config.yaml")

and a Data value should naturally be serializable back to YAML:

yaml.dump(config)

This relationship should be sufficiently direct that HQL does not need an intermediate generic object model merely to represent a YAML document.

JSON can fit the same tree model, although YAML is the more direct model for HyperMarkDown because HMD already uses YAML for its optional authored block.

This makes Data the common structural substrate between HQL, HyperMarkDown metadata, configuration, and external structured information.

Paths¶

Because Data is a tree, path navigation is one of its fundamental operations.

Given:

database:
  primary:
    host: db.example.org
    port: 5432

the natural HQL expression is:

data.database.primary.host

Dynamic access can use the same tree without pretending the keys were statically declared fields:

data["database"]["primary"]["host"]

The precise path API remains part of the HQL language design, but path traversal follows directly from the nature of Data.

Data and HQL extensions¶

Data is also an important boundary type for the HQL extension architecture.

An extension frequently needs to expose structured information without introducing an entire hierarchy of new HQL types.

For example:

postgres.describe("users") : Data

Extension-contributed card metadata works the same way, which is why extensions can enrich a card without the HQL core gaining a document type per extension. See hql/extensions.

or:

git.metadata(...) : Data

or:

http.get(...).yaml() : Data

The extension can preserve the source structure as Data. Stronger HQL types can be added where there is a genuine semantic reason for them.

This keeps the HQL core type system small while still allowing extensions to expose arbitrarily rich structured information.

Data and knowledge¶

Data describes structure, not meaning.

For example:

alice:
  daughter: bob

is a Data tree.

Its presence alone does not necessarily establish a semantic HQL relation such as:

daughterOf(Alice, Bob)

That requires interpretation.

This creates an important boundary in HQL:

Data       structured information
Knowledge  semantic information

Data can contain a representation of knowledge, just as YAML can contain a representation of almost anything. But the tree itself is not automatically the semantic model.

A parser, schema, declaration, or extension can interpret Data and produce concepts, relations, propositions, evidence, or other HQL values.

Data as a central HQL type¶

Data should therefore be treated as one of HQL's core types rather than as a convenience container.

It is the native type of arbitrary hierarchical structured information.

A HyperMarkDown header is Data.

YAML documents are Data.

Most untyped or loosely typed metadata is Data.

Extension-defined structured metadata can be Data.

Configuration and external JSON/YAML-like payloads can be Data.

The defining idea is not that Data is a generic object or Map<String, Any>. There is no Any: every position in the tree is a leaf, a sequence, or a string-keyed map.

The defining idea is:

Data is a structured value whose shape is a tree.

That tree can later be queried, transformed, validated, interpreted, or converted into stronger HQL types without losing the original structure.

See doc-type for the document header, card-type for the knowledge-bearing kind of document, pipes for composition, and hql/knowledge for the semantic side of the Data/Knowledge boundary.