HXL-Data-Science-file-formats

HDP Declarative Programming conventions, early draft

This is an draft. See https://github.com/EticaAI/HXL-Data-Science-file-formats/issues/16.

---

Trivia:

• HDP naming:
  • HDP = ‘HDP Declarative Programming’ is the default name.
    • When in doubt (or you or your tools can’t detect intent of use in immediate context) this is a good way to call it. See Wikipedia for Declarative programming and Recursive acronym
  • HDP = ‘Humanitarian Declarative Programming’ could be one way to call when the intent of the moment is strictly humanitarian.
    • The definition of humanitarian is out of scope.

---

• High level goals
• Conteiner format
  • Exchange of commands with foreign interface
  • Conventions for identifiers (key names)
    • Tokens meaning on keys
      • The << >>, <<! !>>, , <<!! !!>>, , <<!!! !!!>>
• Collaborative work
• Security considerations

---

High level goals

TODO: draft this

Conteiner format

1. The preferred container format is YAML (https://yaml.org/)
   1. Platforms specific features, like the functionality of loading native directly in HDP tools:
      1. Should be ignored in existing YAML files
      2. Should NOT be exported with HDP+YAML format, even if well behaved HDP+YAML parsers would likely to ignore.
2. JSON, in special for compatibility with other tools or for machine-to-machine HDP exchange or for specific low level calls, can be used as additional conteiner format.

Exchange of commands with foreign interface

1. The recommended way of exposing HDP functionality to low level data processing that already not native on HDP is both exporting and accepting JSON format, without the full semantics of HDP (like translations of vocabulary)
   1. [INFORMATIONAL] Ansible uses JSON for data exchange of plugins.

Conventions for identifiers (key names)

Tokens meaning on keys

# Quick example 1
- hsilo:           # hsilo is an verb, without additional tokens
    linguam: LAT   # linguam is also an verb, LAT is an value

Note: Remove \ from \{\{ and \}\} if this document is renderized with them.

• HDP internationalized vocab: No token
  • Description:
    • No token means an word that is perfectly valid HDP vocab that should be able to be translated to EVERY know natural language enabled by as it is.
  • Examples:
    • hsilo_{linguam: LAT}
      • silo_{linguam: ENG|SPA|LAT|POR}
    • linguam_{linguam: LAT}
      • language _{linguam: ENG}
      • язык_{linguam: RUS}
• HDP internationalized vocab with metadata: [ ^prefix, [[ ^prefix , ] ^suffix, ]] ^suffix
  • Description:
    • TODO
  • Examples:
    • TODO
  • Notes:
    • Reserved undocumented token.
• HDP vocab with digitaly signed values: { ^prefix, \{\{ ^prefix , } ^suffix, \}\} ^suffix
  • Description:
    • Token to explicitly mention that the content of this key are digitally signed
    • Removing the [{, \{\{, \}\}, }], the resulting term must be perfectly valid HDP vocab that should be able to be translated to EVERY know natural language enabled by as it is.
  • Notes:
    • For sake of simplicity, if an upper key already is digitally signed, interfaces do not need to show these tokens if all contents, with exception of [<, <<, >>, >>], are signed.
    • When tokes [(, ((, ), ))] are present, the explicit use of this token us not yet defined, since this meas that an outisde program also need to be evaluated.
      • Maybe this with [(!( term-here ))] this notation could be good enough?
  • Examples:
    • \{\{datum\}\}_{linguam: LAT} ^{contents digitally signed and recently verified}
    • {?{datum}?}_{linguam: LAT} ^{contents digitally signed, but not verified yet or this computer can’t do automated verification}
    • {!!!{datum}!!!}_{linguam: LAT} ^{contents digitally signed, but with 1) error, 2) explicitly untrusted or 3) this environment do not tolerate {?{datum}?}}
• Eval: ( ^prefix, (( ^prefix , ) ^suffix, )) ^suffix
  • Notes:
    • Reserved undocumented token.
• Comment: < ^prefix, << ^prefix , > ^suffix, >> ^suffix
  • Description:
    • This key can be used as comment
    • Do not evaluate. Can be safely ignored contents. Do not try to guess the core mesage
  • Examples:
    • <<COMMENT>>, <ENG<COMMENT>>, <POR<COMENTARIO>>

The << >>, <<! !>>, , <<!! !!>>, , <<!!! !!!>>

Draft from v0.8.3

• << >>, <(.*)< >(.*)>
  • For sake of convention, we could use this more for user-related comments
  • the (.*) means that something can be put betwen <</>>, like <POR<Meu comentário aqui em Português>>
• <<! !>>
  • When processing HDP files, internal functions often need to pass information that is essential for functionality (or at least to speed up). This extra ! could be used for this type of internal comment
    • Maybe we could allow user expose this with ‘verbose’ parameters?
• <<!! !!>>, <<!!! !!!>>
  • Similar to <<! !>>, but for higher level of information that is not necessary for end user, but may be for who would work with the library
    • Maybe we could allow user expose this with ‘debug’ parameters?

Issue: ! character is sometimes used to mean ‘encrypted’, so maybe we should choose another character

Collaborative work

See collaborative-work.md

Security considerations

See security-considerations.md

This site is open source. Improve this page.