The DB-All.e Python bindings provide 2 levels of access to a DB-All.e database: a complete API similar to the Fortran and C++ API, and a high-level API called volnd that allows to automatically export matrices of data out of the database.
Contents
The 'dballe' module has a few global methods:
- describe_level(ltype1=None, l1=None, ltype2=None, l2=None)
- Return a string description for a level
- describe_trange(pind=None, p1=None, p2=None)
- Return a string description for a time range
- var(varcode[, default])
- Query the DB-All.e variable table returning a Var, optionally initialized with a value
- varinfo(varcode)
- Query the DB-All.e variable table returning a Varinfo
and several classes, documented in their own sections.
Var holds a measured value, which can be integer, float or string, and a `wreport.Varinfo`_ with all available information (description, unit, precision, ...) related to it.
Var objects can be created from a `wreport.Varinfo`_ object, and an optional value. Omitting the value creates an unset variable.
Examples:
v = wreport.Var(table["B12101"], 32.5) # v.info returns detailed informations about the variable in a Varinfo object. print("%s: %s %s %s" % (v.code, str(v), v.info.unit, v.info.desc))
- code
- variable code
- info
- Varinfo for this variable
- isset
- true if the value is set
- enq()
enq() -> str|float|long
get the value of the variable, as int, float or str according the variable definition
- enqc()
enqc() -> str
get the value of the variable, as a str
- enqd()
enqd() -> float
get the value of the variable, as a float
- enqi()
enqi() -> long
get the value of the variable, as an int
- format(default='')
format(default="") -> str
return a string with the formatted value of the variable
- get(default=None)
get(default=None) -> str|float|long|default
get the value of the variable, with a default if it is unset
Varinfo object holds all possible information about a variable, such as its measurement unit, description and number of significant digits.
Varinfo objects cannot be instantiated directly, and are created by querying `wreport.Vartable`_ objects.
- bit_len
- number of bits used to encode the value in BUFR
- bit_ref
- reference value added after scaling, for BUFR decoding
- code
- variable code
- desc
- description
- len
- number of significant digits
- scale
- scale of the value as a power of 10
- type
- return a string describing the type of the variable (string, binary, integer, decimal)
- unit
- measurement unit
Collection of Varinfo objects indexed by WMO BUFR/CREX table B code.
A Vartable is instantiated by the name (without extension) of the table file installed in wreport's data directory (normally, /usr/share/wreport/):
table = wreport.Vartable("B0000000000000023000") print(table["B12101"].desc) for i in table: print(i.code, i.desc)
- get_bufr
- Vartable.get_bufr(basename=None, originating_centre=0, originating_subcentre=0,
- master_table_number=0, master_table_version_number=None, master_table_version_number_local=0) -> wreport.Vartable
Look up a table B file using the information given, then load BUFR information from it.
You need to provide either basename or master_table_version_number.
- get_crex
- Vartable.get_crex(basename=None, edition_number=2,
- originating_centre=0, originating_subcentre=0, master_table_number=0, master_table_version_number=None, master_table_version_number_bufr=None, master_table_version_number_local=0) -> wreport.Vartable
Look up a table B file using the information given, then load CREX information from it.
You need to provide either basename or master_table_version_number or master_table_version_number_bufr.
- load_bufr
Vartable.load_bufr(pathname) -> wreport.Vartable
Load BUFR information from a Table B file and return it as a wreport.Vartable.
- load_crex
Vartable.load_crex(pathname) -> wreport.Vartable
Load CREX information from a Table B file and return it as a wreport.Vartable.
- pathname
- name of the table
A record holds a number of key->value mappings, similar to a dict.
Keys are strings, from a fixed set of keywords defined in DB-All.e and documented in the Fortran API documentation, plus all varcodes from the DB-All.e BUFR/CREX B table. Using unknown keys raises an exception.
Values are values internally stored inside dballe.Var objects, with a dballe.Varinfo corresponding to the key. Record's mapping interface directly exposes the variable values, but the dballe.Var objects are still accessible using Record.var(key).
A Record is used to make queries to the database, and read results.
When creating a new record, keyword arguments can be passed and they are set as if Record.update(**kwargs) had been called.
When setting items, either via __setitem__ and via update(), there are special keys that allow to set many items in a single call:
# Set year, month, day, hour, min, sec rec["datetime"] = None rec["datetime"] = datetime.datetime(2015, 06, 21, 12, 0, 0) # Set yearmin..secmin and yearmax..secmax rec["datetime"] = (None, None) rec["datetime"] = (datetime.datetime(2014), None) rec["datetime"] = (None, datetime.datetime(2015)) rec["datetime"] = (datetime.datetime(2014), datetime.datetime(2015)) # Set leveltype1, l1, leveltype2, l2 rec["level"] = None rec["level"] = (103, 2000) rec["level"] = (103, 2000, 103, 10000) # Set pindicator, p1, p2 rec["trange"] = None rec["trange"] = (205, 0, 10800)
Examples:
rec = Record(lat=44.05, lon=11.03, B12101=22.1) # Key lookup returns the value of the corresponding variable print(rec["lat"], rec["B12101"]) # Use .var() to get the variable itself; note that it can return # None if the value is not set print(rec.var("lat").info.desc, rec.var("B12101").info.desc) # Iterating a record iterates all keys for key in rec: print(key, rec.get(key, "undefined"), rec.var(key).info.desc) # Use .vars() to get a list of only the varcode keys for varcode in rec.vars(): print(varcode, rec.get(varcode, "undefined"), rec.var(varcode).info.desc)
- clear()
- remove all data from the record
- clear_vars()
- remove all variables from the record, leaving the keywords intact
- copy()
- return a deep copy of the Record
- date_extremes()
- (deprecated) get two datetime objects with the lower and upper bounds of the datetime period in this record
- get(key, default=None)
- lookup a value, returning a fallback value (None by default) if unset
- items()
- return a list with all the (key, value) tuples set in the Record.
- key(key)
- (deprecated) return a dballe.Var from the record, given its key.
- keys()
- return a list with all the keys set in the Record.
- set_from_string(str)
- (deprecated) set values from a 'key=val' string
- set_station_context()
- (deprecated) set the date, level and time range values to match the station data context
- update(**kwargs)
- set many record keys/vars in a single shot, via kwargs
- var(key)
- return a dballe.Var from the record, given its key.
- varitems()
- return a list with all the (key, dballe.Var) tuples set in the Record.
- vars()
- (deprecated) return a sequence with all the variables set on the Record. Note that this does not include keys.
a DB is used to access the database.
Its members are:
- attr_insert_data
- Insert new attributes into the database
- attr_insert_station
- Insert new attributes into the database
- attr_query_data
- Query attributes
- attr_query_station
- Query attributes
- attr_remove_data
- Remove attributes
- attr_remove_station
- Remove attributes
- insert_data
- Insert data values in the database
- insert_station_data
- Insert station values in the database
- query_station_data
- Query the station variables in the database; returns a Cursor
- remove_station_data
- Remove station variables from the database
- attr_insert(varcode, attrs, reference_id=None, replace=True)
- Insert new attributes into the database
- attr_remove(varcode, reference_id, attrs=None)
- Remove attributes
- connect(dsn, user='', password='')
- (deprecated) Create a DB connecting to an ODBC source
- connect_from_file(filename)
- Create a DB connecting to a SQLite file
- connect_from_url(url)
- Create a DB as defined in an URL-like string
- connect_test()
- Create a DB for running the test suite, as configured in the test environment
- disappear()
- Remove all our traces from the database, if applicable.
- export_to_file(query, format, filename, generic=False)
- Export data matching a query as bulletins to a named file
- insert(record, can_replace=False, can_add_stations=False)
- (deprecated)Insert a record in the database
- is_url(string)
- Checks if a string looks like a DB url
- load(file)
load(fp, encoding=None)
Load a file object in the database. An encoding can optionally be provided as a string ("BUFR", "CREX", "AOF"). If encoding is None then load will try to autodetect based on the first byte of the file.
- query_attrs(varcode, reference_id, attrs=None)
- Query attributes
- query_data(query)
- Query the variables in the database; returns a Cursor
- query_stations(query)
- Query the station archive in the database; returns a Cursor
- query_summary(query)
- Query the summary of the results of a query; returns a Cursor
- remove(query)
- Remove variables from the database
- reset([repinfo_filename])
- Reset the database, removing all existing Db-All.e tables and re-creating them empty.
- vacuum()
- Perform database cleanup operations
Examples:
# Connect to a database and run a query db = dballe.DB.connect_from_file("db.sqlite") query = dballe.Record(latmin=44.0, latmax=45.0, lonmin=11.0, lonmax=12.0) # The result is a dballe.Cursor, which can be iterated to get results as # dballe.Record objects. # The results always point to the same Record to avoid creating a new one # for every iteration: if you need to store them, use Record.copy() for rec in db.query_data(query): print rec["lat"], rec["lon"], rec["var"], rec.var().format("undefined") # Insert 2 new variables in the database rec = dballe.Record( lat=44.5, lon=11.4, level=(1,), trange=(254,), date=datetime.datetime(2013, 4, 25, 12, 0, 0), B11101=22.4, B12103=17.2, ) db.insert(rec)
a Cursor is the result of database queries. It is generally not used explicitly and just iterated, but it does have a few members:
- attr_query
- Query attributes for the current variable
- remaining
- number of results still to be returned
- query_attrs(attrs=None)
- Query attributes for the current variable
volnd is an easy way of extracting entire matrixes of data out of a DB-All.e database.
This module allows to extract multidimensional matrixes of data given a list of dimension definitions. Every dimension definition defines what kind of data goes along that dimension.
Dimension definitions can be shared across different extracted matrixes and multiple extractions, allowing to have different matrixes whose indexes have the same meaning.
This example code extracts temperatures in a station by datetime matrix:
query = dballe.Record() query["var"] = "B12001" query["rep_memo"] = "synop" query["level"] = (105, 2) query["trange"] = (0,) vars = read(self.db.query(query), (AnaIndex(), DateTimeIndex())) data = vars["B12001"] # Data is now a 2-dimensional Masked Array with the data # # Information about what values correspond to an index in the various # directions can be accessed in data.dims, which contains one list per # dimension with all the information corresponding to every index. print("Ana dimension is", len(data.dims[0]), "items long") print("Datetime dimension is", len(data.dims[1]), "items long") print("First 10 stations along the Ana dimension:", data.dims[0][:10]) print("First 10 datetimes along the DateTime dimension:", data.dims[1][:10])
This is the list of dimensions supported by dballe.volnd:
- AnaIndex
Index for stations, as they come out of the database.
The constructor syntax is: AnaIndex(shared=True, frozen=False, start=None).
The index saves all stations as AnaIndexEntry tuples, in the same order as they come out of the database.
- NetworkIndex
Index for networks, as they come out of the database.
The constructor syntax is: NetworkIndex(shared=True, frozen=False, start=None).
The index saves all networks as NetworkIndexEntry tuples, in the same order as they come out of the database.
- LevelIndex
Index for levels, as they come out of the database
The constructor syntax is: LevelIndex(shared=True, frozen=False), start=None.
The index saves all levels as dballe.Level tuples, in the same order as they come out of the database.
- TimeRangeIndex
Index for time ranges, as they come out of the database.
The constructor syntax is: TimeRangeIndex(shared=True, frozen=False, start=None).
The index saves all time ranges as dballe.TimeRange tuples, in the same order as they come out of the database.
- DateTimeIndex
Index for datetimes, as they come out of the database.
The constructor syntax is: DateTimeIndex(shared=True, frozen=False, start=None).
The index saves all datetime values as datetime.datetime objects, in the same order as they come out of the database.
- IntervalIndex
Index into equally spaced points in time, starting at start, with a point every step time.
Index points are at fixed time intervals, and data is acquired in one point only if it is within a given tolerance from the interval.
The constructor syntax is: IntervalIndex(start, step, tolerance=0, end=None, shared=True, frozen=False).
start is a datetime.datetime object giving the starting time of the time interval of this index.
step is a datetime.timedelta object with the interval between sampling points.
tolerance is a datetime.timedelta object specifying the maximum allowed interval between a datum datetime and the sampling step. If the interval is bigger than the tolerance, the data is discarded.
end is an optional datetime.datetime object giving the ending time of the time interval of the index. If omitted, the index will end at the latest accepted datum coming out of the database.
The data object used by AnaIndex is:
- AnaIndexEntry
AnaIndex entry, with various data about a single station.
- It is a named tuple of 4 values:
- id: station id
- lat: latitude
- lon: longitude
- ident: mobile station identifier, or None
The extraction is done using the dballe.volnd.read function:
- read(cursor, dims, filter=None, checkConflicts=True, attributes=None)
cursor is a dballe.Cursor resulting from a dballe query
dims is the sequence of indexes to use for shaping the data matrixes
filter is an optional filter function that can be used to discard values from the query: if filter is not None, it will be called for every output record and if it returns False, the record will be discarded
checkConflicts tells if we should raise an exception if two values from the database would fill in the same position in the matrix
attributes tells if we should read attributes as well: if it is None, no attributes will be read; if it is True, all attributes will be read; if it is a sequence, then it is the sequence of attributes that should be read.
The result of dballe.volnd.read is a dict mapping output variable names to a dballe.volnd.Data object with the results. All the Data objects share their indexes unless the xxx-Index definitions have been created with shared=False.
This is the dballe.volnd.Data class documentation:
- Data
Container for collecting variable data. It contains the variable data array and the dimension indexes.
If v is a Data object, you can access the tuple with the dimensions as v.dims, and the masked array with the values as v.vals.
The methods of dballe.volnd.Data are:
- append(self, rec)
Collect a new value from the given dballe record.
You need to call finalise() before the values can be used.
- appendAttrs(self, rec, codes=None)
Collect attributes to append to the record.
You need to call finalise() before the values can be used.
- finalise(self)
- Stop collecting values and create a masked array with all the values collected so far.