jazz_elements Namespace Reference

The namespace for Jazz Utils, Blocks, Kinds, Tuples, Containers, etc. More...

Data Structures
class	Block
	A block is a moveable BlockHeader followed by a Tensor and a StringBuffer. More...
struct	BlockHeader
	Header for a Movable Block (Tensor, Kind or Tuple) or a Dynamic Block (Index) More...
class	Channels
	Channels: A Container doing block transactions across media (files, folders, shell, http urls and zeroMQ servers) More...
class	ConfigFile
	A configuration file as a key/value store. More...
class	Container
	Container: A Service to manage Jazz blocks. All Jazz blocks are managed by this or a descendant of this. More...
struct	EntityKeyHash
	EntityKeyHash: A record containing separate hashes for entity and key. More...
struct	ItemHeader
	Header for an item (of a Kind or Tuple) More...
struct	JazzLmdbOptions
	All the necessary LMDB options (a binary representation of the values in the config file) More...
class	Kind
	Kind: A type definition for Jazz Blocks and Tuples. More...
struct	Locator
	Locator: A minimal structure to define the location of resources inside a Container. More...
class	Logger
	A simple logger. More...
struct	NameUse
	NameUse: A pair of Name and number of times the name is used. More...
struct	ParseNextStateLUT
	A lookup table for all the possible values of a char mapped into an 8-bit state. More...
struct	ParseStateTransition
	A way to build constants defining the transition from one state to the next via a regex. More...
class	Persisted
	Persisted: A Service to manage data objects in LMDB. More...
struct	PutBuffer
	A structure keep state inside a put callback. More...
struct	QueueEnt
	QueueEnt: An entity to store a priority queue. More...
class	Service
	A Jazz Service is a globally instanced configurable object that may allocate RAM. More...
struct	Socket
	A structure to hold a single pipeline. More...
struct	StaticBlockHeader
	A Binary Compatible BlockHeader without Index (and therefore constructors/destructors) More...
struct	StoredTransaction
	An internal (for Container) Transaction with pointers for a deque. More...
struct	StringBuffer
	Structure at the end of a Block, initially created with init_string_buffer() More...
union	Tensor
	A tensor of cell size 1, 4, 8 or sizeof(BlockHeader) More...
struct	TensorDim
	The dimension of a tensor. More...
struct	Transaction
	Transaction: A wrapper over a Block that defines the communication of a block with a Container. More...
class	Tuple
	Tuple: A Jazz Block with multiple Tensors. More...
class	Volatile
	Volatile: A Service to manage data objects in RAM. More...
struct	VolatileTransaction
	VolatileTransaction: A Transaction-descendant wrapper over a Block for Volatile blocks. More...

Typedefs
typedef std::map< int, const char * >	AttributeMap
	An stdlib map to store all the attributes of a Block at the same time used by the some Block methods.
typedef class Block *	pBlock
	A (forward defined) pointer to a Block.
typedef std::map< int, std::string >	MapIS
	A map for defining http config names.
typedef std::map< std::string, Index >	ConnMap
	A structure holding connections.
typedef std::vector< uint8_t >	GetBuffer
	A structure to share with the libcurl get callback.
typedef GetBuffer *	pGetBuffer
	A pointer to a GetBuffer.
typedef PutBuffer *	pPutBuffer
	A pointer to a PutBuffer.
typedef std::map< std::string, Socket >	PipeMap
	A structure holding pipeline.
typedef std::map< int, int >	MapII
	A map for defining http config ports.
typedef unsigned int	MHD_StatusCode
	A proper type for specifying http status codes.
typedef Channels *	pChannels
	A pointer to a Channels.
typedef ParseStateTransition	ParseStateTransitions[NUM_STATE_TRANSITIONS]
typedef std::atomic< int32_t >	Lock32
	An atomically increased (via fetch_add() and fetch_sub()) 32 bit signed integer to use as a lock.
typedef class Container *	pContainer
	A (forward defined) pointer to a Container.
typedef std::map< std::string, pContainer >	BaseNames
	A map of names for the containers (or structure engines like "map" or "tree" inside Volatile).
typedef Transaction *	pTransaction
	A pointer to a Transaction.
typedef struct ExtraLocator *	pExtraLocator
	A pointer to an ExtraLocator.
typedef Locator *	pLocator
	A pointer to a Locator.
typedef StoredTransaction *	pStoredTransaction
	A pointer to a StoredTransaction.
typedef std::map< std::string, int >	MapSI
	An internal map for managing dimension parsing.
typedef Kind *	pKind
	A pointer to a Kind object.
typedef std::map< std::string, MDB_dbi >	DBImap
	The lmdb MDB_dbi handles for each source.
typedef MDB_txn *	pMDB_txn
	A pointer to a MDB_txn structure which is what mdb_txn_begin() returns.
typedef Persisted *	pPersisted
	A pointer to a Persisted object.
typedef Tuple *	pTuple
	A pointer to a Tuple object.
typedef std::chrono::steady_clock::time_point	TimePoint
	A time point stored as 8 bytes.
typedef char	Name[NAME_SIZE]
	A short identifier used in Blocks, Containers and API.
typedef Name *	pName
	A pointer to a Name.
typedef char *	pChar
	A pointer to a char buffer.
typedef uint16_t	ff_fp16
	A future (C++23 declares std::float16_t) 16-bit floating point placeholder for float16 and bfloat16.
typedef std::set< std::string >	Dimensions
	An set::set with the dimension names returned by kind.dimensions()
typedef std::map< std::string, std::string >	Index
	An Index kept in RAM by Volatile implemented as an stdlib map (string, string)
typedef BlockHeader *	pBlockHeader
	A pointer to a BlockHeader.
typedef StaticBlockHeader *	pStaticBlockHeader
	A pointer to a StaticBlockHeader.
typedef StringBuffer *	pStringBuffer
	A pointer to a StringBuffer.
typedef int	StatusCode
	Type returned by the Service API.
typedef int	TenBitIntLUT[TENBITS_LUT_SIZE]
	A lookup table for all the possible results of a TenBitsAtAddress() call -> integer.
typedef void *	TenBitPtrLUT[TENBITS_LUT_SIZE]
	A lookup table for all the possible results of a TenBitsAtAddress() call -> pointer.
typedef ConfigFile *	pConfigFile
	A pointer to a ConfigFile object.
typedef Logger *	pLogger
	A pointer to a Logger object.
typedef Service *	pService
	A pointer to a Service object.
typedef struct VolatileTransaction *	pVolatileTransaction
	A pointer to a Transaction-descendant wrapper over a Block for Volatile blocks.
typedef std::map< uint64_t, pVolatileTransaction >	HashVolXctMap
	HashVolXctMap: A map from hashes to pointers to VolatileTransaction.
typedef std::map< uint64_t, QueueEnt >	HashQueueEntMap
	HashQueueEntMap: A map from hashes to QueueEnt.
typedef std::map< EntityKeyHash, pVolatileTransaction >	EntKeyVolXctMap
	EntKeyVolXctMap: A map from (entity, key) hashes to pointers to VolatileTransaction.
typedef std::map< uint64_t, NameUse >	HashNameUseMap
	HashNameUseMap: A map from hashes to Name and number of times the name is used.
typedef std::string *	pString
	A pointer to an std::string.
typedef Volatile *	pVolatile
	Pointer to Volatile.

Functions
size_t	get_callback (char *ptr, size_t size, size_t nmemb, void *container)
	A callback for libCURL GET.
size_t	dev_null (char *_ignore, size_t size, size_t nmemb, void *_ignore_2)
	A callback for libCURL GET to ignore all the blocks sent by the server in PUT and DELETE calls.
size_t	put_callback (char *ptr, size_t size, size_t nmemb, void *container)
	A callback for libCURL PUT.
void	compile_next_state_LUT (ParseNextStateLUT lut[], int num_states, ParseStateTransition trans[])
double	R_ValueOfNA ()
bool	FileExists (const char *file_name)
	Check if a file exists.
char *	ExpandEscapeSequences (char *buff)
	Expand escaped strings at run-time.
pid_t	FindProcessIdByName (const char *name)
	Find the pid of a process given its name.
uint64_t	MurmurHash64A (const void *key, int len)
	MurmurHash2, 64-bit versions, by Austin Appleby.
std::string	CleanConfigArgument (std::string s)
	Remove quotes and (space and tab) outside quotes from a string.
int	TenBitsAtAddress (const char *str)
	Get ten bits taking the least significant 5 of the first two characters of a string.
int64_t	elapsed_mu_sec (TimePoint big_bang)
	Return server running time in microseconds as a 64 bit integer.
bool	valid_name (pChar p_name)
	Check if a name is valid (Without using the regex).

Variables
char	NA [8] = NA_AS_TEXT
	The text "NA" as a char array.
char	ESCAPE_LOW_ASCII [8] = {"abtnvfr"}
	The letters that can be escaped.
char	DEF_INT8_FMT [8] = {"%hhu\0"}
	The default format for int8_t.
char	DEF_INT32_FMT [8] = {"%i\0"}
	The default format for int32_t.
char	DEF_INT64_FMT [8] = {"%lli\0"}
	The default format for int64_t.
char	DEF_FLOAT32_FMT [8] = {"%.9e\0"}
	The default format for float.
char	DEF_FLOAT64_FMT [8] = {"%.18e\0"}
	The default format for double.
char	DEF_FLOAT_TIME [24] = {"%Y-%m-%d %H:%M:%S"}
	The default format for time_t.
uint32_t	F_NA_uint32
	A binary exact copy of F_NA.
uint64_t	R_NA_uint64
	A binary exact copy of R_NA.
int	LOCATOR_SIZE [3] = {SHORT_NAME_SIZE - 1, NAME_SIZE - 1, NAME_SIZE - 1}
	The size for each section.
ParseStateTransitions	state_tr
ParseNextStateLUT	parser_state_switch [MAX_NUM_PSTATES]
ConfigFile	CONFIG (JAZZ_DEFAULT_CONFIG_PATH)
	The configuration file.
Logger	LOGGER (CONFIG, "LOGGER_PATH")
	The logger.
float	F_NA = nanf("")
	NaN in single.
double	R_NA = R_ValueOfNA()
	NaN in double (binary R-compatible)
ff_fp16	F16_NA = {0x7e00}
	FLOAT16_NA: 0x7e00 is the IEEE 754 half-precision floating point representation of NaN.
ff_fp16	BF16_NA = {0x7fc0}
	BFLOAT16_NA: 0x7fc0 is the Brain Floating Point, half-precision representation of NaN.

Detailed Description

The namespace for Jazz Utils, Blocks, Kinds, Tuples, Containers, etc.

This namespace includes utilities, types, constants and structures used by Block, all different block-based data and code structures (Tuple, Kind, Snippet, ...). All the Services used to allocate/store and communicate blocks (Channels, Volatile, Persisted, ..).

Typedef Documentation

MHD_StatusCode

typedef unsigned int jazz_elements::MHD_StatusCode

A proper type for specifying http status codes.

Before libmicrohttpd (somewhere between > 0.9.66-1 and <= 0.9.72-2) changed MHD_Result to an enum, MHD_Result was (improperly) used to define HTTP responses. That ended-up badly on newer versions, since it was passed to a MHD_queue_response() and stopped working as it became an enum.

This triggered the need, for clarity reasons only, to introduce a new type, MHD_StatusCode to refer to HTTP responses.

See: https://www.iana.org/assignments/http-status-codes/http-status-codes.xhtml

ParseStateTransitions

typedef ParseStateTransition jazz_elements::ParseStateTransitions[NUM_STATE_TRANSITIONS]

A vector of StateTransition.l This only runs once, when construction the API object, initializes the LUTs from a sequence of StateTransition constants in the source of api.cpp.

Name

typedef char jazz_elements::Name[NAME_SIZE]

A short identifier used in Blocks, Containers and API.

Names are used in may contexts including: identifying a Container in an API query, a Container inside another Container, a Block in a Container, a field in a Tuple or a Kind and some API query arguments (e.g., as_json).

Names are vanilla ASCII NAME_LENGTH long string starting with a letter and containing just letters, numbers and the underscore. They can be validated using the function valid_name() or the regex REGEX_VALIDATE_NAME.

HashVolXctMap

typedef std::map<uint64_t, pVolatileTransaction> jazz_elements::HashVolXctMap

HashVolXctMap: A map from hashes to pointers to VolatileTransaction.

This map allows locating entity root VolatileTransactions for creating and destroying entities.

HashQueueEntMap

typedef std::map<uint64_t, QueueEnt> jazz_elements::HashQueueEntMap

HashQueueEntMap: A map from hashes to QueueEnt.

This map allows locating entity root VolatileTransactions for creating and destroying queues.

EntKeyVolXctMap

typedef std::map<EntityKeyHash, pVolatileTransaction> jazz_elements::EntKeyVolXctMap

EntKeyVolXctMap: A map from (entity, key) hashes to pointers to VolatileTransaction.

This map allows locating any nodes.

HashNameUseMap

typedef std::map<uint64_t, NameUse> jazz_elements::HashNameUseMap

HashNameUseMap: A map from hashes to Name and number of times the name is used.

This map allows doing the reverse conversion to a hash() function finding out the hashed names.

Function Documentation

get_callback()

size_t jazz_elements::get_callback	(	char *	ptr,
		size_t	size,
		size_t	nmemb,
		void *	container
	)

A callback for libCURL GET.

Parameters

ptr	The incoming data chunk.
size	Size in whatever_units.
nmemb	whatever_unit size in bytes.
container	A pointer owned by the caller. (A GetBuffer in this case.)

Returns

The number of bytes processed.

(see https://curl.haxx.se/libcurl/c/CURLOPT_WRITEFUNCTION.html)

dev_null()

size_t jazz_elements::dev_null	(	char *	_ignore,
		size_t	size,
		size_t	nmemb,
		void *	_ignore_2
	)

A callback for libCURL GET to ignore all the blocks sent by the server in PUT and DELETE calls.

Parameters

_ignore	The incoming data chunk.
size	Size in whatever_units.
nmemb	whatever_unit size in bytes.
_ignore_2	A pointer owned by the caller. (A GetBuffer in this case.)

Returns

The number of bytes processed.

(see https://curl.haxx.se/libcurl/c/CURLOPT_WRITEFUNCTION.html)

put_callback()

size_t jazz_elements::put_callback	(	char *	ptr,
		size_t	size,
		size_t	nmemb,
		void *	container
	)

A callback for libCURL PUT.

Parameters

ptr	The incoming data chunk.
size	Size in whatever_units.
nmemb	whatever_unit size in bytes.
container	A pointer owned by the caller. (A GetBuffer in this case.)

Returns

The number of bytes processed.

(see https://curl.haxx.se/libcurl/c/CURLOPT_READFUNCTION.html)

compile_next_state_LUT()

void jazz_elements::compile_next_state_LUT	(	ParseNextStateLUT	lut[],
		int	num_states,
		ParseStateTransition	trans[]
	)

The ParseNextStateLUT compiler

Parameters

lut	The array to fill with the LUT.
num_states	The number of states in the LUT.
trans	The array of transitions to use to fill the LUT.

(This is only used to create constants used by parsers.)

R_ValueOfNA()

double jazz_elements::R_ValueOfNA ( )

inline

Constants for the types of cells in a Tensor

Returns

An R-compatible numeric NA.

FileExists()

bool jazz_elements::FileExists

(

const char *

file_name

)

Check if a file exists.

Parameters

file_name

The file name.

Returns

True if the file exists

ExpandEscapeSequences()

char * jazz_elements::ExpandEscapeSequences

(

char *

buff

)

Expand escaped strings at run-time.

Public Domain by Jerry Coffin.

Interprets a string in a manner similar to that the compiler does string literals in a program. All escape sequences are longer than their translated equivalent, so the string is translated in place and either remains the same length or becomes shorter.

April 17 tests built and compliant with:

\a 07 Alert (Beep, Bell) (added in C89)[1] \b 08 Backspace \f 0C Form-feed \n 0A Newline (Line Feed); see notes below \r 0D Carriage Return \t 09 Horizontal Tab \v 0B Vertical Tab \\ 5C Backslash \' 27 Single quotation mark \" 22 Double quotation mark \? 3F Question mark (used to avoid trigraphs) \nnn The byte whose numerical value is given by nnn interpreted as an octal number \xhh The byte whose numerical value is given by hh… interpreted as a hexadecimal number

https://en.wikipedia.org/wiki/Escape_sequences_in_C

Parameters

buff	The string to be expanded.

Returns

The expanded string.

FindProcessIdByName()

pid_t jazz_elements::FindProcessIdByName

(

const char *

name

)

Find the pid of a process given its name.

It does NOT find itself as it is intended to find other processes of the same name.

Parameters

name	The name of the process as typed on the console. In case parameters were given to start the process, it is just the left part before the first space.

Returns

The pid of the process if found, 0 if not.

MurmurHash64A()

uint64_t jazz_elements::MurmurHash64A	(	const void *	key,
		int	len
	)

MurmurHash2, 64-bit versions, by Austin Appleby.

(from https://sites.google.com/site/murmurhash/) a 64-bit hash for 64-bit platforms

All code is released to the public domain. For business purposes, Murmurhash is under the MIT license.

The same caveats as 32-bit MurmurHash2 apply here - beware of alignment and endian-ness issues if used across multiple platforms.

Parameters

key	address of the memory block to hash.
len	Number of bytes to hash.

Returns

64-bit hash of the memory block.

CleanConfigArgument()

std::string jazz_elements::CleanConfigArgument

(

std::string

s

)

Remove quotes and (space and tab) outside quotes from a string.

Removes space and tab characters except inside a string declared with a double quote '"'. After doing that, it removes the quotes. This is used by ConfigFile and has obvious limitations but is used for its simplicity and easily predictable results.

Parameters

s	Input string

Returns

String without space or tab.

TenBitsAtAddress()

int jazz_elements::TenBitsAtAddress ( const char *  str )

inline

Get ten bits taking the least significant 5 of the first two characters of a string.

Parameters

str	A string with at least two characters.

Returns

The ten bits as an integer.

Warning: No pointer validation or length check. Never use on nullptr or "".

elapsed_mu_sec()

int64_t jazz_elements::elapsed_mu_sec ( TimePoint  big_bang )

inline

Return server running time in microseconds as a 64 bit integer.

Parameters

big_bang

The primordial event (== Jazz server start ;)

Returns

The time elapsed in microseconds

valid_name()

bool jazz_elements::valid_name ( pChar  p_name )

inline

Check if a name is valid (Without using the regex).

Parameters

p_name

The name to be checked

Returns

True if valid

Variable Documentation

state_tr

ParseStateTransitions jazz_elements::state_tr

The parser logic defined in terms of transitions between states.

parser_state_switch

ParseNextStateLUT jazz_elements::parser_state_switch[MAX_NUM_PSTATES]

The parser logic defined as a LUT (initialized by compile_next_state_LUT()).