web logs and remote_ips

[r_layden]

Hi

I'm looking for advice about working with web_logs and in particular how we might track students' usage through remote_ips.

I'm currently maintaining a small server running MariaDB which uses the dap client to retrieve CD2 tables.

Firstly - how feasible is it to download all of your web_logs?
We've been a Canvas institution for 5 years so that's going to be a heck of a lot of records.

Is there a way to download a small snapshot into my database using the initdb/syncdb command?
I suppose I could just do a 'dap --incremental' query and then upload the files to the database?
But then how would I get subsequent updates? 

Also - specifically about the format of the remote_ip field, if I choose to get an incremental csv snapshot the ip address is formatted in a readable format (i.e. it looks like an IP address) 
But, on my MariaDB table I just seem to see the binary blob (?) e.g.  ▒5j#

Is that because I'm using MariaDB I wonder?  Or is there something easy that I'm missing.

Many thanks for your time

Rich

[LeventeHunyadi · Accepted solution]

Currently, DAP (a.k.a. CD 2) API does not serve web logs dating back to several years. Only the most recent few months can be exported, even with a snapshot query.

IPv4/6 addresses are stored as type inet in PostgreSQL, and type BINARY(16) in MySQL and Microsoft SQL Server (which don't have a dedicated type for IPv4/6 addresses). You may need to use conversion functions to extract integer values of the components. For example, the IPv4 address 192.168.0.1 would be packed into four bytes, each of which would be stored as a binary character (hence the blob-like display). Formats TSV and CSV represent IP addresses in their standard notation.

[r_layden]

Thank you so much 
Much appreciated

Rich