Users missing from USER_DIM

[smitsrr]

We are working with the canvasCLI tool to integrate canvas activity into our data warehouse. 

We have joined our student SIS data to canvas successfully using SIS_USER_ID in pseudonym_dim, however, when we try to get their GLOBA_CANVAS_ID from user_dim, there are no records for about 1/2 of our users. 

I'm joining pseudonym_dim.user_id = user_dim.id

As additional evidence of a problem with missing data, we have roughly 440k unique records in pseudonym_dim, and only 220k unique records in user_dim. 

Is there a reason users are not included in user_dim? We have good evidence that students enrolled in courses are those not included. 

[stuart_ryan]

Hi  @smitsrr ,

That is an awesome question! I am not entirely sure on this one myself, so I am going to share this into https://community.canvaslms.com/groups/big-data?sr=search&searchId=afb10f18-514a-4465-83cc-e631ccafa... as there may be some good minds in there that can help you out, especially on the Canvas Data front.

Hope that helps!

Stuart

[smitsrr · Accepted solution]

Just to update followers, the issue was actually that our warehousing software was scrubbing records with negative ID's, which had the effect of removing approximately 1/2 the rows. 

We learn something new every day!

[stuart_ryan]

Hi  @smitsrr ,

Awesome to hear that you got this resolved! Also, thank you for coming back and posting the resolution for future people to find if they have the same issue!

Cheers,

Stuart