ERL 12- If you build it, will they come?

Kate Silton, North Carolina A&T State University
May Yan, Ryerson University

Assessment of their institutions’ implementations of Summon

Assessment in general:

  • What are your goals? Who are your stakeholders?
  • Scope should reflect resources available to carry out project
  • Assessment projects should be specific and targeted

North Carolina A&T: Assessing Summon’s freshman year

  • How frequently does it deliver the full text of the article?
  • Will students stop using other databases?
  • Who is using this tool and how are they using it?

1. Delivery of full text

  • Did a search, tested first 100 results — did it work?
  • About 60% got to full text (originally it used link resolver, after Summon implemented direct links  it got better)
  • Tested out some limiters as well – newspaper full text hard to get to, but sometimes the metadata for these are bad and hard for link resolvers to work with
  • ebooks were difficult as well as first, got better with direct linking
  • delivery of full text by vendor: some vendors have worked better than others – major problems with Gale links until direct linking

2. Impact on e-resources usage

  • Proquest usage bumped up, Springer too, others look like they went the other way

3. Who’s using Summon?

  • admin console provides lots of data about users
  • most visits from IE9 on Windows
  • surprising number of users on dialup
  • lots of users using it on campus, and via proxy server
  • top queries: electrical engineering, blank search, usual first year Qs

Next steps:

  • systematic plan to deal with broken links
  • continue to monitor usage
  • follow up usage analytics with a survey or focus groups

 Summon evaluation at Ryerson University

  • goals: evaluate investment in Summon, determine where to devote resources to maximize investment, increase awareness of Summon
  • methodology: questionnaire (whole university), focus groups (students) and interviews (faculty/researchers)
  • wanted study to be repeatable over the years
  • How is Summon being used and who is using it?
  • are they satisfied with it? Rate ease of use? any usability issues?
  • Has there been enough promotion of Summon? Info on how to use it? What other info needs to be provided?
  • Contrast experience between user groups: undergrads/grads & faculty, novice vs. experience library users, disciplines of study, Summon and other projects (Google Scholar, subject indexes)


  • online survey using Survey Monkey
  • 10-15 questions
  • open and close ended questions
  • all faculty, staff and students
  • advertised on library homepage, posters, LCD screens, mailing list to all students and faculty (in phase 2 only)
  • incentive: bookstore GCs in first phase, 3 iPads in second phase
  • Inductive process relies on getting ‘good’ feedback from open ended Qs in questionnaire and focus group/interview — more challenging to carry out consistently over time
  • two questionnaires because the first was intended as a learning experience, so they’d know how to do it best in phase 2 (marketing too)

Focus groups and interviews currently happening

Response rates & results:

  • phase 1, 191 responses, over half had used it and 75% said they’d use it again
  • phase 2, 6344 responses – direct email to mailing list worked best
  • for last academic search, 60% had used Summon – mostly for writing essays, also used web search engines, library website, professors and instructors, other websites (not search engines)
  • How easy was it to use? Mostly very easy
  • Easy to find what they needed? Moderately easy category was the highest
  • How satisfied? Very-moderately satisfied

Concluding thoughts:

  • Many evaluation strategies, might want to use combinations to get a more complete view
  • expect that it will take time for a shift in user behavior and plan evaluation accordingly

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s