There has been an explosion of growth in using AI, data science, and machine learning in all aspects of our daily life. There is a global competition among governments, industry, and academic institutions to lead research and development in this area. This paper discusses a novel multidisciplinary graduate education and research program at our institution to help develop a trained workforce to meet the demands required to understand and develop AI, data science and machine learning technologies. The program brings together faculty and students in engineering, computer science, and social science to build a traineeship program where cohort teams study fundamental and applied data science research, using compact modules across courses to personalize instruction and prepare each trainee with skills tailored to their prior experience and future career goals.

This workshop aims to address the common bad practices and habits that novice software engineering students develop and how educators can detect them and guide students towards better practices before they become ingrained. The workshop explored this by soliciting contributions of bad practices educators have observed in teaching introductory software engineering. Contributions were workshopped to develop a compendium of bad practices, along with proposed remediations and subsequently shared with the software engineering education community. The workshop concluded with recommendations for using the compendium to improve software engineering education.

Extract Method is one of the most widely used refactoring patterns. So far, low quality of source code has been regarded as an indicator for Extract Method opportunities. However, recent studies showed that there is no clear relationship between source code quality and Extract Method. Change metrics can be indicators for Extract Method because the characteristics of software evolution strongly affect software quality. However, there has been no study that investigated the relationship between change metrics and Extract Method. In this study, we conducted two studies investigating the relationship between Extract Method and change metrics. As a result, we found that (1) change metrics have a clear relationship with Extract Method and (2) both product and change metrics are necessary to recommend candidates for Extract Method with high accuracy.

Context/ Background: The Mission Design and Navigation Software (MDN) Group at the Jet Propulsion Laboratory (JPL) develops and continuously maintains software systems critical for NASA deep space missions. Given limited budgets, staffing resources, and a time critical need for repair or enhancement, there is an ever-present temptation to sacrifice quality for higher productivity or slip release target to ensure better quality. We have learned that poor management of this increases risk of mission failure. As a result, our process must be both highly productive and maintain high quality (e.g. reliability, maintainability, usability). Inspired by the "quality is free" paradigm, we have instituted a set of "Rapid Release" maintenance process policies and measures aimed to continually manage productivity and quality. Six Rapid Release polices were established from well-known engineering principles and best practices to address specific issues of concern encountered in the development phase. However, due to the critically of our systems, we must have objective assurance that our developers are following the six policies and that they are demonstratively effective in addressing the areas of concern.

We present our findings after two years of experience involving three instructors using an "athletic" approach to software engineering education (AthSE). Co-author Johnson developed AthSE in 2013 to address issues he experienced teaching graduate and undergraduate software engineering. Co-authors Port and Hill subsequently adapted the original approach to their own software courses. AthSE is a pedagogy in which the course is organized into a series of skills to be mastered. For each skill, students are given practice "Workouts" along with videos showing the instructor performing the Workout both correctly and quickly. Unlike traditional home-work assignments, students are advised to repeat the Workout not only until they can complete it correctly, but also as quickly as the instructor. In this experience report we investigate the following question: how can software engineering education be redesigned as an athletic endeavor, and will this provide more efficient and effective learning among students and more rapidly lead them to greater competency and confidence?

Context: The Mission Design and Navigation Software Group at the Jet Propulsion Laboratory (JPL) maintains mission critical software systems. We have good empirical data and models for maintenance demand---when defects will occur, how many and how severe they will be, and how much effort is needed to address them. However, determining the level of staffing needed to address maintenance issues is an ongoing challenge and is often done ad-hoc. There are two common strategies are (1) reactive - add/remove staff as needed to respond to maintenance issues, and (2) capacitive - retain a given staff size to address issues as they occur, proactively address issues and prevent defects.

Context: There is a pervasive feeling that somehow software assurance is important. For example, national space agencies, such as the National Aeronautics and Space Administration (NASA) and the Japan Aerospace Exploration Agency (JAXA), require assurance for their critical software systems. Their assurance effort goes beyond testing, and includes activities such as process compliance checks, artifact audits, and traceability validation. These activities can be costly and their benefits, particularly in managing risk, are poorly understood. This inevitably leads to tough questions concerning value such as "Is assurance worth doing?" and "How much assurance do we actually need?"

Context: At the Mission Design and Navigation Software Group at the Jet Propulsion Laboratory we make use of finite exponential based defect models to aid in maintenance planning and management for our widely used critical systems. However a number of pragmatic issues arise when applying defect models for a post-release system in continuous use. These include: how to utilize information from problem reports rather than testing to drive defect discovery and removal effort, practical model calibration, and alignment of model assumptions with our environment.

Students with little professional software development experience typically have low intrinsic motivation and beyond achieving a good grade, low extrinsic motivation to study and appreciate the value of software engineering curricula. Unlike other subjects, introductory software engineering instructors exert a great deal of effort justifying and motivating their course topics. Since 2006 the University of Hawaii has been developing a series of “early awareness” engagements within an introductory Systems Analysis and Design course designed to foster intrinsic and extrinsic motivation and orient students to value learning software engineering. Measuring motivation and perceived value is difficult, but there are key indicators to determine the impact of our improvements such as higher course evaluations, greater class attendance, and increased positive feedback from students and employers. Our results show that these key indicators have improved since introducing early engagements. Furthermore, students like this approach, value the course more, do better quality work, and evaluate the course more positively. Longitudinal follow-ups indicate greater interest and success in pursuing software engineering related careers. This paper shares the details of our early awareness engagements, how they are applied in the classroom, some of our experiences in using them, and evidence that they have a positive impact. Our goal is to provide specific and practical means that instructors can use immediately to improve the perceived value of software engineering for novice students prior to or while they study it.

A software release is a decision to deliver code to an organization outside of the development team usually for testing or operational purposes. For critical systems this can be a risky decision where failure to pass a test or holding up the project schedule can have a major impact. The release decision is primarily based on the understanding on the level of quality the software currently has (be it high quality, low quality, or unknown). But for large, complex systems, determining the level of quality with high confidence is a challenge. A poor understanding of the confidence in the quality level increases decision risk leading potentially to a bad release decision that possibly could have been avoided had the confidence in the quality been better known. Certification of release readiness attempts to address this risk by building confidence in the quality level. But this comes at a cost and the relationship between certification and decision risk reduction has not been well understood. This work describes our experience investigating the value of certification and our efforts to improve the mandated software readiness certification record (SRCR) process. A well known critical system at JPL is used as a case study to exemplify this effort.

It is often claimed Real-Client Real-Project (RCRP) courses are important providers of industry relevant experience and skills to students. How do we know this is so? We cannot prove this or improve RCRP industry relevance without tangible evidence. Here we suggest that the degree an industry partner is willing to accept technology transfer for technologies used within an RCRP course is a strong indicator of relevance. There must be common challenges between RCRP and industry software development for which technology proven in the classroom will also be relevant in industry. We describe our experiences with using an RCRP parallel to the TAME technology transfer model to assess the degree of willingness for technology transfer facilitated by technology use in RCRP courses. We find benefits from realizing envisaged synergies between software engineering research, education, and technology transfer to industry. We further note how the willingness for technology transfer indicator is useful for aligning software engineering courses for industry relevance.

Often repositories of systems engineering artifacts at NASA's Jet Propulsion Laboratory (JPL) are so large and poorly structured that they have outgrown our capability to effectively manually process their contents to extract useful information. Sophisticated text mining methods and tools seem a quick, low-effort approach to automating our limited manual efforts. Our experiences of exploring such methods mainly in three areas including historical risk analysis, defect identification based on requirements analysis, and over-time analysis of system anomalies at JPL, have shown that obtaining useful results requires substantial unanticipated efforts - from preprocessing the data to transforming the output for practical applications. We have not observed any quick 'wins' or realized benefit from short-term effort avoidance through automation in this area. Surprisingly we have realized a number of unexpected long-term benefits from the process of applying text mining to our repositories. This paper elaborates some of these benefits and our important lessons learned from the process of preparing and applying text mining to large unstructured system artifacts at JPL aiming to benefit future TM applications in similar problem domains and also in hope for being extended to broader areas of applications.

Insufficient risk analysis often leads to software system design defects and system failures. Assurance of software risk documents aims to increase the confidence that identified risks are complete, specific, and correct. Yet assurance methods rely heavily on manual analysis that requires significant knowledge of historical projects and subjective, perhaps biased judgment from domain experts. To address the issue, we have developed RARGen, a text mining-based approach based on well-established methods aiming to automatically create and maintain risk repositories to identify usable risk association rules (RARs) from a corpus of risk analysis documents. RARs are risks that have frequently occurred in historical projects. We evaluate RARGen on 20 publicly available e-service projects. Our evaluation results show that RARGen can effectively reason about RARs, increase confidence and cost-effectiveness of risk assurance, and support difficult-to-perform activities such as assuring complete-risk identification.

This paper augments Boehm-Turner's model of agile and plan-based software development augmented with an AI search algorithm. The AI search finds the key factors that predict for the success of agile or traditional plan-based software developments. According to our simulations and AI search algorithm: (1) in no case did agile methods perform worse than plan-based approaches; (2) in some cases, agile performed best. Hence, we recommend that the default development practice for organizations be an agile method. The simplicity of this style of analysis begs the question: why is so much time wasted on evidence-less debates on software process when a simple combination of simulation plus automatic search can mature the dialog much faster?

In this paper, we address the efficacy and pragmatics of mixing two primary strategies for requirements prioritization in order to incorporate the benefits of both plan-based (PB) and agile development methods while avoiding their drawbacks. As it is intractable to directly study the performance of strategies on real projects, we conducted a comprehensive empirically based simulation under a variety of conditions of requirements dynamism, project size, and duration. Simulation results suggest that a mixed strategy for requirements prioritization seems to work best in all but cost for typical levels of dynamism on average. Our findings also indicate that, as theorized, PB and agile strategies perform well within opposite extremes of dynamism. However, they do not outperform the mixed strategies even within their home grounds – that is large and complex systems with stable requirements for PB, and small and dynamic projects for agile methods. Given the unknown, unknowable, or variable nature of dynamism and the dramatic differences in effectiveness for agile and PB strategies under extreme development scenarios, a mixed strategy appears to yield the best results overall. We introduce two mixed strategies – simply adding cost–benefit (CB) to the agile approach, and a more sophisticated ‘hybrid’ (HY) approach that modulates development iteration size to maximize the expected CB for each iteration. We propose a step-by-step method to implement this HY strategy. We provide a structured analysis of the benefits and assumptions of agile and PB requirements prioritization methods (e.g., Pareto optimization), and outline a framework for analyzing and assessing the effectiveness of strategies including several new metrics. This research can furthermore serve as a framework for future validation of the proposed mixed strategies using actual software projects.

The contributions of Nancy R. Mead have greatly shaped what software engineering education and training (SWEET) is today. This paper contains three tributes to Dr. Mead, each focusing on a different aspect of her legacy. The first tribute provides an overall view of her work in the SWEET field. The second essay describes her research contributions in SWEET. The final section discusses her role in the development of this conference and the challenges of building upon the solid foundations that Nancy Mead has provided the field.

Nancy Mead has been expounding the virtues of requirements prioritization, especially in the area of information security, for over a decade. She advocates avoiding purely informal ad-hoc approaches and suggests a prioritization strategy based on both cost of implementing requirements and the expected value if implemented. Due to critical schedule and resource constraints, we have found that requirements prioritization is a key risk management practice for real project for real clients based courses. This work describes our experiences applying Nancy Meadpsilas requirement prioritization works within such courses and how we educate students on the importance of utilizing this for managing requirements implementation risk in student projects within three universities for both graduate and undergraduates.

Software cost model research results depend on model accuracy criteria such as MMRE and PRED. Despite criticism, MMRE has emerged as the de facto standard criterion. Many alternatives have been proposed and studied, surprisingly however PRED, the second most popular criterion, has not been extensively studied. This work attempts to fill this gap in the literature and expand the understanding and use of evaluation criterion in general. The majority of this work is empirically based, applying MMRE and PRED to a number of COCOMO model variations with respect to a simulated data set and four publicly available cost estimation data sets. We replicate a number of results based on MMRE and extend them to PRED. We study qualities of MMRE and PRED as sample estimator statistics for parameters of a cost model error distribution. Standard error is used to ensure greater confidence in replicated and new results based on sample data.

Agile and traditional plan-based approaches to software system development both agree that prioritizing requirements is an essential activity. They differ in basic strategy - when to prioritize, to what degree, and how to guide implementation. As with many software engineering methods, verifying the benefit of following a particular approach is a challenge. Industry and student/classroom based experimental studies are generally impractical to use for large numbers of controlled experiments and benefits are difficult to measure directly. We use simulation to validate the fundamental, yet typically intangible benefits of requirements prioritization strategies. Our simulation is directly based on detailed empirical studies of agile and plan-based requirements management studies. Our simulation shows, as many have claimed, that an agile strategy excels when requirements are highly volatile, whereas a plan-based strategy excels when requirements are stable, and that there exist mixed strategies that are better than either for typical development efforts.

Over the past four years the University of Hawaii has been steadily moving towards an "active learning" approach to software engineering education by replacing passive slide-show lectures with a series of short lectures followed by in-class exercises, enabled though the use of wireless networked laptops. In this paper we report on our experiences and early results with this approach. The results suggest that students like this approach more, do better, and evaluate the courses more positively.

Outsourcing and off shoring are inevitable buzz words today. Outsourcing refers to a firm sending its work to be done by another firm. Off shoring refers to work done at a geographical location far removed from the firm's location. The firm doing the job at the remote location could be a branch of the parent firm or an outsourced vendor / partner. In any case the business drivers are cost reduction, access to skilled resources, complementary process capabilities, supplementary domain / technology competencies and 24x7 customer servicing ability.

An IV&V activity plan describes what assurance activities to perform, where to do them, when, and to what extent. Meaningful justification for an IV&V budget and evidence that activities performed actually provide high assurance has been difficult to provide from plans created (generally ad hoc) by experts. JAXA now uses the "strategic IV&V planning and cost model" to addresses these issues and complement expert planning activities. This research presents a grounded empirical study that compares plans generated by the strategic model to those created by experts on several past IV&V projects. Through this research, we found that the model generated plan typically is a superset of the experts ' plan. We found that experts tended to follow the most cost-effective route but had a bias in their particular activity selections. Ultimately we found increased confidence in both expert and model based planning and now have new tools for assessing and improving them.

In this paper we show how Theory-W and the WinWin requirements negotiation approach have influenced software engineering education in a number of institutions worldwide. After a short summary of Theory-W and the WinWin negotiation model, processes, and tools the paper discusses how students can benefit from WinWin in their software engineering education. We explore different options for teaching the approach and present concrete examples and experiences from different universities

Fred Brooks' seminal book, "The Mythical Man-Month" (MMM) is a firmly established classic in software engineering. Many of us feel compelled to use this work to help our students appreciate and put into practice the fundamental software engineering concepts contained between its covers. This often amounts to using "passive" lesson plans such as required readings followed by lectures and exams; these rarely fully satisfy our learning objectives. Rather, students often have mixed reactions to MMM with the result that it has little impact on their attitudes and practices, both in and out of the classroom. This paper outlines a more active approach to incorporating MMM into the classroom, one that we have refined over 6 years, at multiple universities and in both graduate and undergraduate courses. It includes learning objectives, a lesson plan, sample materials, an implementation discussion, and an evaluation of the approach's impact.

Summary form only given, as follows. The complete presentation was not made available for publication as part of the conference proceedings. The workshop has three objectives: To address the question: what constitutes research in software engineering education (SEE)? What areas of SEE needs to be systematically investigated? Methods of course delivery? Industrial needs? Curriculum improvement? Furthermore, how should the result of such investigation to be evaluated and most effectively organized? To address the question: what are the best outlets, beyond CSEE&T, for publishing the result of research in SEE? In particular, does the SEE community need its own SEE journal? To discuss synergy between research in SEE and research in related disciplines. What learning theories, teaching frameworks, and training practices from other disciplines can be transitioned into the SEE? We will discuss each of the above items in turn. Furthermore, we will review the researchers guide prepared for this conference. Participants will be asked to suggest improvements to it and to comment on how it has affected the quality and rigor of the research presented at the conference.

COCONUT calibrates effort estimation models using an ex-haustive search over the space of calibration parameters in a COCOMO I model. This technique is much simpler than other effort estimation method yet yields PRED levels com-parable to those other methods. Also, it does so with less project data and fewer attributes (no scale factors). How-ever, a comparison between COCONUT and other methods is complicated by differences in the experimental methods used for effort estimation. A review of those experimental methods concludes that software effort estimation models should be calibrated to local data using incremental hold-out (not jack knife) studies, combined with randomization and hypothesis testing, repeated a statistically significant number of times.

Despite the widespread availability of software effort estimation models (e.g. COCOMO [2], Price-S [12], SEER-SEM [13], SLIM [14]), most managers still estimate new projects by extrapolating from old projects [3, 5, 7]. In this delta method, the cost of the next project is the cost of the last project multiplied by some factors modeling the difference between old and new projects [2].Delta estimation is simple, fast, and best of all, can take full advantage of local costing information. However delta estimation fails when the experience base (the old projects) can not be extrapolated to the new projects. Previously [10], we have shown that for a set of NASA projects, delta estimation would usually fail since most of the features and coefficients of the learned model vary wildly across sub-samples of the training data. In that prior work, no solution was offered for this problem.Here, we offer a solution and report the results of experiment with feature subset selection (FSS) and extrapolation. FSS methods are usually assessed via the mean change in model performance. However, as shown below, FSS can significantly reduce the variance as well. Hence, FSS should be routinely used in cost estimation.Our results should stop the trend in the effort modeling community of continually adding to the number of features in a model in order to improve estimation performance. Here we show that there are benefits in intelligently subtracting model features.

We describe how empirical research performed in the context of a software engineering project course can provide results useful to both students in later offerings of the course and to industry. The secondary purpose is to encourage the formation of an effort to share ideas on classroom-based research, to perform research collaboratively, and to share results with both the academic and industrial software engineering communities.

The primary purpose of this tutorial is to show participants how empirical research performed in the context of a Software Engineering project course can provide results useful to both students in later offerings of the course and to industry. The tutorial's secondary purpose is to encourage the formation of an effort to share ideas on classroom-based research, to perform research collaboratively, and to share results with both the academic and industrial software engineering communities.

Workshop Summary The purpose of this workshop is to present an in-depth overview of USC CS577, and to start the processes of: • eliciting from faculty at a variety of colleges and universities the different needs and resource constraints to which they would be subject were they to port CS577 to their home institutions • planning the types of adjustments that can be made to CS577 to meet various sets of constraints

Software development is one of the most economically critical engineering activities. It is unsettling, therefore, that regularly published analyses reveal that the percentage of projects that fail, by coming in far over budget or far past schedule, or by being cancelled with significant financial loss, is considerably greater in software development than in any other branch of engineering. The reason is that successful software development requires expertise in both state of the art (software technology) and state of the practice (software development process). It is widely recognized that failure to follow best practice, rather than technological incompetence, is the cause of most failures. It is critically important, therefore, that (i) computer science departments be able assess the quality of the software development process component of their curricula and that industry be able to assess the efficacy of SPI (software process improvement) efforts. While assessment instruments/tools exist for knowledge of software technology, none exist for attitude toward, knowledge of, or ability to use, software development process. We have developed instruments for measuring attitude and knowledge, and are working on an instrument to measure ability to use. The current version of ATSE, the instrument for measuring attitude toward software engineering, is the result of repeated administrations to both students and software development professionals, post-administration focus groups, rewrites, and statistical reliability analyses. In this paper we discuss the development of ATSE, results, both expected an unexpected, of recent administrations of ATSE to students and professionals, the various uses to which ATSE is currently being put and to which it could be put, and ATSE's continuing development and improvement.

Workshop Summary The purpose of this workshop is to provide university faculty and software development professionals with a background on work already done in developing a tool (ATSE) for measuring students’ attitudes to software process, and to involve them in the exploration of its use and further development. The outcome of this workshop will include guidelines for application of ATSE and other instruments within academic computer science programs and in industrial SPI efforts. The workshop will also involve the participant in the evolution of the various instruments.

It is widely acknowledged that group or team projects are a staple of undergraduate and graduate software engineering courses. Such projects provide students with experiences that better prepare them for their careers, so teamwork is often required or strongly encouraged by accreditation agencies. While there are a multitude of educational benefits of group projects, they also pose considerable challenge in fairly and accurately discerning individual contribution for evaluation purposes. Issues, approaches, and best practices for evaluating individual contribution are presented from the perspectives of the University of Kentucky, University of Ottawa, University of Southern California, and others. The techniques utilized within a particular course generally are a mix of (1) the group mark is everybody's mark, (2) everybody reports what they personally did, (3) other group members report the relative contributions of other group members, (4) pop quizzes on project details, and (5) cross-validating with the results of individual work.

Most projects commit to a set of required features and (at best) a most-likely budget and schedule for developing them. This means that, even before the changes start coming, there is roughly a 50% chance that the most-likely budget and schedule are insufficient, and the project is headed for an overrun. Planning for change in a development project is essential. But how much should be invested in architectural flexibility to accommodate this? Too little will incur a high risk of costly late changes and architecture breakage; too much may not leave enough time to implement a sufficient set of critical capabilities. We have been using and refining a model based approach to assist in determining an appropriate degree of architectural flexibility by introducing a modularity factor for the software architecture based on the core capabilities and a set of anticipated changes. This experience has helped us identify the critical success factors for strategically applying architectural flexibility within tight constraints such as cost, quality, or a fixed schedule. We elaborate the critical success factors, present a case study of their application, and their relation to recent research results in such areas as strategic design.

The University of Southern California (USC) teaches a two-semester real-client project course as a core course in USC's MSCS-SE (Master of Science in Computer Science and Software Engineering) degree program. The course has evolved rapidly, each year introducing many changes in order to satisfy the course stakeholder's win conditions. The course has also been our primary experimental testbed for evolving our MBASE (Model-Based Architecting and Software Engineering) model integration framework. In turn, this framework, along with the CRESST (Center for Research on Evaluation, Standards and Student Testing) cognitive demands analysis has served as an effective means of managing the course's rapid evolution. A further test of the framework has been its application to undergraduate software engineering project courses at other institutions. This paper provides a description and examples of USC's experience in constructing and evolving a family of software engineering project courses based on the MBASE software engineering model integration framework. A discussion of course needs, use of software engineering models, a description of MBASE, use of the CRESST model of learning objectives, course development and experiences are presented. The approach has been used successfully at Columbia University and to a lesser degree at other institutions. Much of the current and historical materials and resources described in this paper are freely available for educational use.

In 1996, the University of Southern California (USC) switched its core two-semester software engineering course from a hypothetical-project, homework-and-exam course based on the Bloom taxonomy of educational objectives (knowledge, comprehension, application, analysis, synthesis and evaluation). The revised course is a real-client team-project course based on the CRESST (Center for Research on Evaluation, Standards and Student Testing) model of learning objectives (content understanding, problem solving, collaboration, communication and self-regulation). We used the CRESST cognitive demands analysis to determine the necessary student skills required for software risk management and the other major project activities, and have been refining the approach over the last four years of experience, including revised versions for one-semester undergraduate and graduate project courses at Columbia University. This paper summarizes our experiences in evolving the risk management aspects of the project courses. These have helped us mature more general techniques, such as risk-driven specifications, domain-specific simplifier and complicator lists, and the SAIV (schedule as an independent variable) process model. The largely positive results in terms of review pass/fail rates, client evaluations, product adoption rates and hiring manager feedback are summarized as well.

Since 1996 the authors have had great success combining software engineering research and software engineering education. This effort introduces new research and experimental areas into the curriculum along with the means to evolve and refine current practices. The curriculum itself becomes a valuable research area with direct applications to real-world practice. The approach in particular tries to deal with four fundamental issues within software engineering and software engineering education. There are Exposing students to a no-surprises full lifecycle software engineering experience. Dealing with the increasing distribution and diversity of students. Providing students a non-trivial real-world customer-based development experience within the constraints of the university. Introducing software engineering research and industry best practices into the curriculum.

A study was commissioned by the Hughes Space and Communications Software Engineering Group to determine the effectiveness of the recent introduction of object-oriented languages, technologies and development methodologies. Of particular interest was any effect on development productivity. Fundamentally, productivity metrics are difficult to apply across non-homogeneous projects and development teams. Furthermore, owing to many uncontrolled variables, such as the lack of a solid control project and non-rigorously collected data, productivity measures alone are were sufficient to determine meaningful results as requested for the study. A new, robust means of comparing non-homogenous development efforts (that does not require a control project) called "efficiency" was introduced and was used to augment the comparative analysis of the projects and address possible concerns with the use of productivity metrics alone. Efficiency measures the actual effort compared to an estimate of that effort with respect to an independent and well-defined baseline - for purposes of the study, this was COCOMO II. Conclusive evidence was found to support the hypothesis that object-oriented languages coupled with object-oriented methods result in greater productivity and efficiency as compared to other efforts. Furthermore, it is concluded that efficiency metrics, along with the non-standard use of COCOMO II, are a meaningful, useful and practical approach to compare development efforts.

One of the difficulties in requirements negotiation is to determine a feasible and mutually satisfactory set of requirements for the developer and the user, a problem related to C.P. Snow's (1959) "Two Cultures" problem. During the last year of our experience with an annual series of digital library projects, we have been experimenting with expectations management and domain specific lists of "simplifiers" and "complicators", as a way to address the "Two Cultures" problem involving librarians and computer scientists. Initial results indicate that the simplifiers and complicators approach successfully reduced the number of projects having serious feasibility problems, and helped manage the expectations of both the developers and the customers/users. We see no obstacles to applying the approach to other domains.