CD2 - Weblogs

[raoshriya96]

I'm trying to extract all clicks to a specific URL from web logs, with user ID-level detail. However, when filtering the logs to that URL, I noticed some users have multiple records (5-10) with timestamps only a few seconds apart. This suggests repeated clicks on the same link within a short span, which seems unlikely. Can you help me understand why this might be happening? Is there noise in the data causing these duplicate records? Additionally, can you provide guidance on how to accurately pull a list of all clicks to the website, including any necessary considerations or filters to apply to weblogs ?

[mclark19]

Weblogs records contain activity in the system, some of which is human-initiated (such as clicking on a link) and some of which is system-internal (stuff going on "under the hood" when e.g., a user clicks on a link). One human action can result in many web_logs records associated with that action. I'll let others weigh in on ways to get useful data out, as we tend to just report on very generally aggregate data, such as unique users (user_id) per day.

[raoshriya96]

Thank you for the info @mclark19 ! When you reference reporting on the user_id, do you take into account any timestamp associated with the user's activity? In other words, how do you determine which record to consider as the actual user click - do you follow the first timestamp per user_id or do you consider other factors to identify the relevant record?

[mclark19]

@raoshriya96 -- Our goal is just to get a very global sense of daily system usage (in terms of number of unique students) over the course of a semester, so we would just do a count for distinct user_id for each day (with some additional filtering to make sure that those user_ids were associated with an active StudentEnrollment in a course for the semester in question). So in that sense, we aren't really targeting any specific record -- the existence of *any* web_logs record associated with a student would include them in the count for that day. This was always intended to be a "back of the napkin" metric as opposed to a definitive count, as the goal was just to create a chart of "unique daily users" across the semester (date on the x-axis, user count on the y-axis). As expected, we see more unique users during the week (trailing off as we get to the end of the week), a drop off on the weekend and during Spring Break, etc. To use an analogy that I wish I could claim was mine (can't remember where I heard it), we are "cutting with a chainsaw" at the end of the day, so we don't really need to "measure with a micrometer" to draw the line beforehand.