ASDF: An Autonomous and Scalable Distributed File System . many of the same goals as previous distributed file systems such as scalability. Distributed File System (DFS) could be a set of consumer and server .. Huang and Huan-Ming Liang, “ASDF: An Autonomous and Scalable Distributed. File. 3 / Abstract. We have simulated SimpleFS, a simple and scalable distributed file system for nowadays highly Other file systems, like TidyFS, ASDF, Ceph, etc, utilize similar concepts but take more .. ASDF: An Autonomous and. Scalable.
|Published (Last):||6 October 2014|
|PDF File Size:||9.21 Mb|
|ePub File Size:||10.85 Mb|
|Price:||Free* [*Free Regsitration Required]|
ASDFthe de facto standard build system for Common Lisp, has been vastly improved between and These and other improvements finally bring Common Lisp up to par with “scripting languages” in terms of ease of writing and deploying portable code that can access and “glue” together functionality from the underlying system or external programs.
We describe the most salient improvements in ASDF and how they enable previously difficult and portably impossible uses of the programming language. It was explicitly designed to allow for high-performance implementations; some of them, ecalable on the application, may rival compiled C programs in terms of speed, usually far ahead sn “scripting” languages and their implementations.
There are over a dozen maintained or unmaintained CL implementations. No single one is at the same time the best, shiniest, leanest, fastest, cheapest, and the one ported to the most platforms. Or you might desire the technical autlnomous or additional libraries from a proprietary implementation. Happily, each implementation provides its own systme and there exist libraries to abstract over the discrepancies between these implementations and provide portable access to threads bordeaux-threadsUnicode support cl-unicode disyributed, a “foreign function dishributed to libraries written in C cffiML-style pattern-matching optimaetc.
A software distribution system, Quicklisp, makes it easy to install hundreds of libraries that use ASDF. The new features in ASDF 3 were only the last missing pieces in this puzzle. The last barrier to making that possible was the lack of a portable way to build and deploy code so a same script can run unmodified for many users on one or many machines using one or many different compilers.
This was solved by ASDF 3. The purpose of a build system is to enable division of labor in software development: ASDF 3 is the latest rewrite of the system. Aside from fixing numerous bugs, it sports a new portability layer UIOP. One can now use ASDF to write Lisp programs that may be invoked from the command line or may spawn external programs and capture their output ASDF can deliver these programs as standalone executable files; moreover the companion script cl-launch see cl-launch can create light-weight scripts that can be run unmodified on many different kinds of machines, each differently configured.
Scalabke features make portable scripting possible. In Code Evolution in a Conservative Communitywe discuss the challenges of evolving a piece of community software, concluding with lessons learned from our experience; these lessons are of general interest to software programmers though the specifics are related to CL. This is the extended version of this article. There again, the specifics will only interest CL programmers, but general lessons can be disstributed that are of general interest to all software practitioners.
Roughly in chronological order, we have the initial successful experiment in Appendix A: Pathnames ; improvements in expressiveness in Appendix D: Failed Attempts at Improvement ; and the bug that required rewriting it all over again in Appendix F: A traverse across the build. All versions of this article are available at http: Top components are called system s in an age-old Lisp tradition, while the bottom ones are source file s, typically written in CL.
In between, there may be a recursive hierarchy of module s that may contain files or other modules and may or may not map to subdirectories. Users may then operate on these components with various build operation s, most prominently compiling the source code operation compile-op and loading the output into the current Lisp image operation load-op.
Several related scalablw may be developed together in the same source code project.
1 Ceph Distributed Storage Sage Weil DreamHost / new dream network April 7, ppt download
Each system may depend on code from other systems, either from the same project or from a different project. Further, each component may explicitly declare a dependency on other components: Below is how the fare-quasiquote system is defined with elisions in a file fare-quasiquote.
It contains three files, packagesquasiquote and pp-quasiquote the. The latter files each depend on the first file, because this former file defines the CL packages Packages are namespaces that contain symbols; they need to be created before the symbols they contain may even be read as valid syntax.
Each CL process has a global flat two-level namespace: Among the elided elements were metadata such as: Notice how the system itself depends-on another system, fare-utilsa collection of utility functions and macros from another project, whereas testing is specified to be done by fare-quasiquote-testa system defined in a different file, fare-quasiquote-test.
This example illustrates the use of modules: The first component is a file package. As you can see, dependencies name sibling components under the same parent system or module, that can themselves be files or modules. The process of building software is modeled as a Directed Acyclic Graph DAG of action s, where each action is a pair of an operation and a component.
The DAG defines a partial order, whereby each action must be perform ed, but only after all the actions it transitively depends-on have already been performed. For instance, in fare-quasiquote above, the loading of the output of compiling quasiquote depends-on the compiling of quasiquotewhich itself depends-on the loading of the output of compiling packageetc.
Importantly, though, this graph is distinct from the preceding graph of components: Unlike its immediate predecessor mk-defsystemASDF makes a plan of all actions needed to obtain an up-to-date version of the build output before it performs these actions.
In ASDF itself, this plan is a topologically sorted list of actions to be performed sequentially: POIU compiles files in parallel on Unix multiprocessors using forkwhile still loading them sequentially into a main image, minimizing latency.
ASDF is an “in-image” build systemin the Lisp defsystem tradition: For better and worse, this notably differs from common practice in most other languages, where the build system is a completely different piece of software running in a separate process.
Of course, a build system could compile CL code in separate processes, for the sake of determinism and parallelism: On the one hand, it minimizes overhead to writing build system extensions.
On the other hand, it puts great pressure on ASDF to remain minimal. Qualitatively, ASDF must be delivered as a single source file and cannot use any external library, since it itself defines the code that may load other files and libraries.
This arguably mattered more in when ASDF was first released and was about a thousand lines long: Byit has grown over ten times in size, but memory sizes have increased even faster. For all these reasons, ASDF follows the minimalist principle that anything that can be provided as an extension should be provided as an extension and left out of the core.
Thus it cannot afford to support a persistence cache indexed by the cryptographic digest of build expressions, or a distributed network of workers, etc. However, these could conceivably be implemented as ASDF extensions.
Most programmers are familiar with C, but not with CL. Note though how these services are factored in very different ways in CL and in C. To build and load software, C programmers commonly use make to build the software and ld.
Additionally, they use a tool like autoconf to locate available libraries and identify their features. ASDF 3 also provides functionality which would correspond to small parts of the libc and of the linker ld. But in other important ways ASDF demonstrates how these C systems have much accidental complexity that CL does away with thanks to better architecture. In C, the many utilities that need a DSL must grow it onerously from scratch; since the domain expert is seldom also a language expert with resources to do it right, this means plenty of mutually incompatible, misdesigned, power-starved, misimplemented languages that have to be combined through an unprincipled chaos of expensive yet inexpressive means of communication.
Lisp provides full introspection at runtime and compile-time alike, as well as a protocol to declare features and conditionally include or omit code or data based on them. In C, people resort to horribly unmaintainable configuration scripts in a hodge podge of shell script, m4 macros, C preprocessing and C code, plus often bits of pythonperlsedetc. ASDF possesses a standard and standardly extensible way to configure where to find the libraries your code depends on, further improved in ASDF 2.
In C, there are tens of incompatible ways to do it, between libtoolautoconfkde-configpkg-configvarious manual. ASDF uses the very same mechanism to configure both runtime and compile-time, so there is only one configuration mechanism to learn and to use, and minimal discrepancy.
There is still discrepancy inherent with these times being distinct: In C, completely different, incompatible mechanisms are used at runtime ld.
Its relative znd is directly related to it being custom made to build CL software only. ASDF 3 could be easily extended to support arbitrary build actions, if there were an according desire. At the other extreme, a build system for CL could have been made that is much simpler and more elegant than ASDFif it could have required software to follow some simple organization constraints, without much respect for legacy code.
The new ASDF 3 design is consistent and general enough that it could conceivably be made to scale, autonnomous that would require a lot of work.
And yet it worked, mostly. As a result, the object model of ASDF became at the same time more powerful, more robust, and simpler to explain. The dark magic of its traverse function is replaced by a well-documented algorithm. Thus, ASDF can now express arbitrary action graphs, and could conceivably be used in the future to build more than just CL programs. The proof diistributed a good design is abd the ease of extending it.
We thus tested our design by adapting the most elaborate existing ASDF extensions to use it. The result was indeed cleaner, eliminating the previous need for overrides that redefined sizable chunks of the infrastructure.
ASDF: An Autonomous and Scalable Distributed File System
Chronologically, however, we consciously started this porting process in interaction with developing ASDF 3, thus ensuring ASDF 3 had all the extension hooks required to avoid redefinitions. See the entire story in Appendix F: Bundle operations create a single output file for an entire system or collection of systems. The most directly user-facing bundle operations are compile-bundle-op and load-bundle-op: Also lib-op links into a library all the object files in a system and dll-op creates a dynamically loadable library out scalbale them.
The above bundle operations also have so-called monolithic variants that bundle all the files in a system and all its transitive dependencies. Bundle operations make delivery of ysstem much easier.
Most CL implementations maintain their own heap with their own garbage collector, and then are able to dump an image of the heap on disk, that can be loaded back in a new process with all the state of the former process.
To build an application, you thus start a small initial image, load plenty of code, dump an image, and there you are. To build an application with ECL or its variant MKCLyou thus link all the libraries and object files together, and call the proper initialization functions in the correct order. Bundle operations are important to deliver software using ECL as a library to be embedded in some C program. Also, because of the overhead of dynamic linking, loading a single object file is preferable to a lot of smaller object files.
In Mayit was generalized to other implementations as the external system asdf-bundle. Note that compile-bundle-opload-bundle-op and deliver-asd-op were respectively called fasl-opload-fasl-op and binary-op in the original asdf-ecl and autonojous successors up until ASDF 3.
They were eventually renamed, with backward compatibility stubs left behind under the old name. After bundle support was merged into ASDF see Bundle Operations aboveit became trivial to implement a new concatenate-source-op operation.