This is a read only archive of pad.okfn.org. See the shutdown announcement for details.

•  Alexandre Hannud Abdo
•  Daniel Mietchen
•  Jenny Molloy

• Konrad Förstner (@konradfoerstner)
• Matthias Fromm (@matthiasfromm, fromm@mfromm.de)
• Catriona MacCallum (@catmacOA)

• showing that open science can be more efficient than traditional approaches under some conditions 
• exploring the space of meaningful conditions in a systematic fashion 
• understanding what outcomes can be properly compared in a practical experiment 
• actually putting open and traditional research to an efficiency test 
• identifying funders that may be interested in supporting such an efficiency test

• Who  could be the research subjects for a study on open science? Is a  particular discipline/country/career stage better suited to such a  study?
• How  could one go about answering the question? Potential research  methodologies and pitfalls, correlated factors and complications to  consider.
• How  can we actually make this happen? What might the funding mechanism be  and are there collaborators or funding agencies that would be  interested?
• An  introduction to how we reached this point - what are the proposed  benefits and problems of open science and what questions should we be  asking more broadly (this session focuses on efficiency)

• 
  
• Disciplines
• In computer science - open source vs. closed source software project
• evaluatoin via hourst spend/time needed - help by external commiters
• Drug development 
• Could not completly open - another layer to anonomyse is needed
• Should be performed in different research field to proove the increase of efficiency for all of them
• ROS (ROS.org) - Robotor Operating System - as an example where an open approach is widely adapted
• Matthew Todd - had a simple setup - testing the activity of compound; did this openly
• Focus maybe only on certain parts of the research cycle e.g. grand application
• would require a larger number of grant applications
• Geography
• Could be done in developing countries

• Career Stage
• For  established scientist (e.g. a professor) has a lower risc and could to  this but would not necessarily have the time; a PhD or Postdoc would 

• Methods
• Challenge/Competition to solve a research question where you can win a price
• As the closed group could have access to the open groups outcome the task would have to differ a litte bit
• Maybe use simple questions like find the optimal concentration of compoind
• Lower price might be an incentive to perform a experiment openly
• Universitis and funder should require openess
• How can we measure the advantage ot the open approach like usuability
• on the long run via data citiations, currently we might miss it
• There is a increains amount of tools forthis in the Digital Humanitiers - this could be used to convetional approaches
• We always need a (closed) control group - in many scenarious this is already available
• Maybe doing a meta study with the open projects that are around
• problem - usually not good control group

• Study Considerations
• 
  
• advantages of open science maybe  harder to test then advantages of open access
• studies/analysis of impact necessary
• mentioning of Human Genome Project ->  huge investments following funding of the project
• efficiency might be a good argument, but how to measure efficiency?
• who can tell how efficient science is, at the moment?
• fund projects that can prove the need for open data, open science - reuse of data, new applications, etc.
• there are criteria besides efficiency, use for society
• the funding landscape
• "how do you provide incentives" - this is something funders can do
• how do we change the incentive system?
• have funders ask the right questions, e.g. was your past research needed?
• increase funders' pressure regarding requirements of openness

• Projects and people
• 
  
• who can do that, who can apply for funds?
• social scientists, economists might be the people to test this, to apply for grants
• citation analysts
• suitable projects
• if open data is useful there should be an impact in papers
• how many papers, how many products, how many actual user resulting from science projects?
• metadata on open access publications, where did innovations, products come from, do they stem from open access publications
• citation analysis
• altmetrics
• look at publication data, citations ?
• fund open review to discover advantages of open science projects
• look for patterns, where changes follow policies
• measure citizen science projects' success

• Potential funders:
• 
  
• see: open access transformation call by DFG, http://www.dfg.de/en/research_funding/announcements_proposals/info_wissenschaft_14_29/index.html, mentioning of "measuring the impact of open-access publications"
• mentionings, examples
• mentioning of HIIG in Berlin, http://www.hiig.de/en/
• http://adrn.ac.uk/
• europe pubmed, http://europepmc.org/

•                                    
• Has this been tried before?
• 
  
• European Comission: open data pilot review
• -they dont know themselves how to evaluate
• -science 2.0 framework: look at evaluation
• -incentives for reuse and efficiency
• -based on how they implemented open access

• Social science

• Viability evidence
• -Identified galaxy times

• What is efficiency?
• 
  
• Two approaches:
• -Efficient for whom
• -the peope, the scientists, the institutions
• -but, public policy can handle incentives to scientists

• Can we assume non-negativity of open?
• -why not making things open, fear
• -overhead: metadata, reproducible outcomes
• -but, it also benefits yourself

• Measurables
• 
  
• Impact of different licenses

• measuring the fear of opening

• Replication
• -assume it is generally a good thing
• -study if it leads to more, better publication
• -compare whether studies have been replicated, for how long, if replications confirm or disprove
• -use licenceses
• -science exchange
• -could be turned into an experiment
• -measure time between publication of work and citation (of some work item) (in open science it could be negative)
• -indexed vs non-indexed open scientific data

• Reuse
• -time for data reuse
• -databases like dryad have both closed and open datasets
• 
  
• -are open scientific works less likely to be lost? (data loss)
• -there is a paper already looking at a baseline for this

• use in other languages

• efficiency of translating scientific resulsts into technology

• Process
• 
  
• -review of past registry
• -social experiments

• take 2 researchers giving a similar task

• -track the open bits of a project in a registry, over multiple projects
• -but how to separate?

• -what can be done, cant be done

• -there are measurables through the process, not only in the outcomes
• -number of collaborators
• -work hour metrics?

• -list of people who applied for a study from a funding body
• -check the outcomes for open and closed

• Bias

• -collaboration with people who are not under the influence of open science advocacy mindset
• -ideal experiment would involve both open and closed science advocates
• -participation of multiple interests in the design

• Considerations

• Our discussion assumes simple measurements vulnerable to variations in context and actors.
• We did not have time to discuss other methods, such as qualitative or mixed methodologies, that might shed light on the bias and counfounders involved.
• Dependencies on what part, tendency, of open science